{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c0bc94f9-f9f8-40c5-9807-a3e21cc4d53f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torchvision import datasets, transforms\n",
    "import time\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ad19817-33da-4b71-ab7e-7dde60c2b6d4",
   "metadata": {},
   "source": [
    "# 一、使用LeNet-5网络结构创建MNIST手写数字识别分类器"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c83d117a-fcf0-456c-820e-0ee5365b7a09",
   "metadata": {},
   "source": [
    "MNIST是一个非常有名的手写体数字识别数据集，训练样本：共60000个，其中55000个用于训练，另外5000个用于验证；测试样本：共10000个。MNIST数据集每张图片是单通道的，大小为28x28."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a61e91f3-cdac-47c0-bb5e-5cfec613b6ff",
   "metadata": {},
   "source": [
    "![](./images/mnist.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f392f9d-a1d4-4e6e-b093-caa587bc6506",
   "metadata": {},
   "source": [
    "## 1.1 下载并加载数据，并做出一定的预先处理"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "129a6ea4-0d2b-428f-b84b-a6ec2e59996f",
   "metadata": {},
   "source": [
    "由于MNIST数据集图片尺寸是28x28单通道的，而LeNet-5网络输入Input图片尺寸是32x32，因此使用transforms.Resize将输入图片尺寸调整为32x32。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4850476e-8c74-4a56-b7fb-263a42a1dd10",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipline_train = transforms.Compose([\n",
    "    #随机旋转图片\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    #将图片尺寸resize到32x32\n",
    "    transforms.Resize((32,32)),\n",
    "    #将图片转化为Tensor格式\n",
    "    transforms.ToTensor(),\n",
    "    #正则化(当模型出现过拟合的情况时，用来降低模型的复杂度)\n",
    "    transforms.Normalize((0.1307,),(0.3081,))    \n",
    "])\n",
    "pipline_test = transforms.Compose([\n",
    "    #将图片尺寸resize到32x32\n",
    "    transforms.Resize((32,32)),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize((0.1307,),(0.3081,))\n",
    "])\n",
    "#下载数据集\n",
    "train_set = datasets.MNIST(root=\"./data\", train=True, download=True, transform=pipline_train)\n",
    "test_set = datasets.MNIST(root=\"./data\", train=False, download=True, transform=pipline_test)\n",
    "#加载数据集\n",
    "trainloader = torch.utils.data.DataLoader(train_set, batch_size=64, shuffle=True)\n",
    "testloader = torch.utils.data.DataLoader(test_set, batch_size=32, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b7235fa-aefa-4a7c-8862-ec563e9279a5",
   "metadata": {},
   "source": [
    "**这里要解释一下Pytorch MNIST数据集标准化为什么是transforms.Normalize((0.1307,), (0.3081,))？**\n",
    "\n",
    "**标准化（Normalization）**是神经网络对数据的一种经常性操作。标准化处理指的是：样本减去它的均值，再除以它的标准差，最终样本将呈现均值为0方差为1的数据分布。\n",
    "\n",
    "神经网络模型偏爱标准化数据，原因是均值为0方差为1的数据在sigmoid、tanh经过激活函数后求导得到的导数很大，反之原始数据不仅分布不均（噪声大）而且数值通常都很大（本例中数值范围是0~255），激活函数后求导得到的导数则接近与0，这也被称为梯度消失。前文已经分析，神经网络是根据函数对权值求导的导数来调整权值，导数越大，调整幅度越大，越快逼近目标函数，反之，导数越小，调整幅度越小，所以说，数据的标准化有利于加快神经网络的训练。\n",
    "\n",
    "除此之外，还需要保持train_set、val_set和test_set标准化系数的一致性。标准化系数就是计算要用到的均值和标准差，在本例中是((0.1307,), (0.3081,))，均值是0.1307，标准差是0.3081，这些系数都是数据集提供方计算好的数据。不同数据集就有不同的标准化系数，例如([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])就是ImageNet dataset的标准化系数（RGB三个通道对应三组系数），当需要将imagenet预训练的参数迁移到另一神经网络时，被迁移的神经网络就需要使用imagenet的系数，否则预训练不仅无法起到应有的作用甚至还会帮倒忙，\n",
    "\n",
    "例如，我们想要用神经网络来识别夜空中的星星，因为黑色是夜空的主旋律，从像素上看黑色就是数据集的均值，标准化操作时，所有图像会减去均值（黑色），如此Imagenet预训练的神经网络很难识别出这些数据是夜空图像！"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "501c973b-5561-4ef7-adfe-078f749560e1",
   "metadata": {},
   "source": [
    "## 1.2 搭建LeNet-5神经网络结构，并定义前向传播的过程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f6802c96-6002-4baf-a6d0-bc5c6932d0cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LeNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(LeNet, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 6, 5) \n",
    "        self.relu = nn.ReLU()\n",
    "        self.maxpool1 = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        self.maxpool2 = nn.MaxPool2d(2, 2)\n",
    "        self.fc1 = nn.Linear(16*5*5, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    " \n",
    "    def forward(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.maxpool1(x)\n",
    "        x = self.conv2(x)\n",
    "        x = self.maxpool2(x)\n",
    "        x = x.view(-1, 16*5*5)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        output = F.log_softmax(x, dim=1)\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "435fed7b-ea7f-40d3-b44b-069b74ecd1bb",
   "metadata": {},
   "source": [
    "## 1.3 将定义好的网络结构搭载到GPU/CPU，并定义优化器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0475276a-c945-4304-aa04-1c9594bea16a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#创建模型，部署gpu\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "model = LeNet().to(device)\n",
    "#定义优化器\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b6f3075-57e1-46cb-ba82-c93e36d07015",
   "metadata": {},
   "source": [
    "## 1.4 定义训练过程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "59c2c551-6a46-49c7-a10b-b85f3329eb12",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_runner(model, device, trainloader, optimizer, epoch):\n",
    "    #训练模型, 启用 BatchNormalization 和 Dropout, 将BatchNormalization和Dropout置为True\n",
    "    model.train()\n",
    "    total = 0\n",
    "    correct =0.0\n",
    "    \n",
    "    #enumerate迭代已加载的数据集,同时获取数据和数据下标\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        inputs, labels = data\n",
    "        #把模型部署到device上\n",
    "        inputs, labels = inputs.to(device), labels.to(device)\n",
    "        #初始化梯度\n",
    "        optimizer.zero_grad()\n",
    "        #保存训练结果\n",
    "        outputs = model(inputs)\n",
    "        #计算损失和\n",
    "        #多分类情况通常使用cross_entropy(交叉熵损失函数), 而对于二分类问题, 通常使用sigmod\n",
    "        loss = F.cross_entropy(outputs, labels)\n",
    "        #获取最大概率的预测结果\n",
    "        #dim=1表示返回每一行的最大值对应的列下标\n",
    "        predict = outputs.argmax(dim=1)\n",
    "        total += labels.size(0)\n",
    "        correct += (predict == labels).sum().item()\n",
    "        #反向传播\n",
    "        loss.backward()\n",
    "        #更新参数\n",
    "        optimizer.step()\n",
    "        if i % 1000 == 0:\n",
    "            #loss.item()表示当前loss的数值\n",
    "            print(\"Train Epoch{} \\t Loss: {:.6f}, accuracy: {:.6f}%\".format(epoch, loss.item(), 100*(correct/total)))\n",
    "            Loss.append(loss.item())\n",
    "            Accuracy.append(correct/total)\n",
    "    return loss.item(), correct/total"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ee02d95-be36-4041-a51f-a90d23e9af52",
   "metadata": {},
   "source": [
    "## 1.5 定义测试过程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ea5af177-b284-40a6-9ef9-cb4da76c1e73",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_runner(model, device, testloader):\n",
    "    #模型验证, 必须要写, 否则只要有输入数据, 即使不训练, 它也会改变权值\n",
    "    #因为调用eval()将不启用 BatchNormalization 和 Dropout, BatchNormalization和Dropout置为False\n",
    "    model.eval()\n",
    "    #统计模型正确率, 设置初始值\n",
    "    correct = 0.0\n",
    "    test_loss = 0.0\n",
    "    total = 0\n",
    "    #torch.no_grad将不会计算梯度, 也不会进行反向传播\n",
    "    with torch.no_grad():\n",
    "        for data, label in testloader:\n",
    "            data, label = data.to(device), label.to(device)\n",
    "            output = model(data)\n",
    "            test_loss += F.cross_entropy(output, label).item()\n",
    "            predict = output.argmax(dim=1)\n",
    "            #计算正确数量\n",
    "            total += label.size(0)\n",
    "            correct += (predict == label).sum().item()\n",
    "        #计算损失值\n",
    "        print(\"test_avarage_loss: {:.6f}, accuracy: {:.6f}%\".format(test_loss/total, 100*(correct/total)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e92fe478-5c21-4be8-9efc-0b09c94532a5",
   "metadata": {},
   "source": [
    "## 1.6 运行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "37fd3f6b-3bca-4824-9756-a1ab93d55cac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start_time 2021-11-27 22:15:09\n",
      "Train Epoch1 \t Loss: 2.312757, accuracy: 12.500000%\n",
      "test_avarage_loss: 0.003749, accuracy: 96.100000%\n",
      "end_time:  2021-11-27 22:15:45 \n",
      "\n",
      "start_time 2021-11-27 22:15:45\n",
      "Train Epoch2 \t Loss: 0.069703, accuracy: 100.000000%\n",
      "test_avarage_loss: 0.002672, accuracy: 97.300000%\n",
      "end_time:  2021-11-27 22:16:20 \n",
      "\n",
      "start_time 2021-11-27 22:16:20\n",
      "Train Epoch3 \t Loss: 0.025734, accuracy: 100.000000%\n",
      "test_avarage_loss: 0.002858, accuracy: 97.130000%\n",
      "end_time:  2021-11-27 22:16:55 \n",
      "\n",
      "start_time 2021-11-27 22:16:55\n",
      "Train Epoch4 \t Loss: 0.155763, accuracy: 93.750000%\n",
      "test_avarage_loss: 0.002237, accuracy: 97.670000%\n",
      "end_time:  2021-11-27 22:17:31 \n",
      "\n",
      "start_time 2021-11-27 22:17:31\n",
      "Train Epoch5 \t Loss: 0.020248, accuracy: 100.000000%\n",
      "test_avarage_loss: 0.002280, accuracy: 97.720000%\n",
      "end_time:  2021-11-27 22:18:07 \n",
      "\n",
      "Finished Training\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAACSCAYAAABsboAjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAASnklEQVR4nO3da3Bc5X3H8e9/L9JKq8UXXSxbki0HYxuHm7GS0JCmw6WFBEpoEjPJtJlppzO8SdqkZdqEJu2knXaaTDtp8qLtDIXQF0mTYOOklGFCmoYppCQEyQ4QYxsM2Fi2ZdkyNrpYl93998XZlVe2jIXZ9Tm7+/vM7Ojcdve/x9ZPzz777HnM3RERkeiKhV2AiIi8NQW1iEjEKahFRCJOQS0iEnEKahGRiFNQi4hEnIJaRCTiFNRS1cxsn5ndHHYdIpWkoBYRiTgFtdQcM2s0s6+b2aHC7etm1ljY12Zmj5rZCTM7bmZPmVmssO/zZnbQzEbNbI+Z3RTuKxEJJMIuQKQCvghcB1wDOPCfwJeAvwTuAQaB9sKx1wFuZuuAzwDvcfdDZtYLxC9u2SLzU4taatHvAn/j7sPufhT4a+BThX0zwHJglbvPuPtTHlzwJgc0AhvMLOnu+9z9lVCqFzmDglpq0Qpgf8n6/sI2gH8A9gI/MrNXzewLAO6+F/gc8GVg2My+a2YrEIkABbXUokPAqpL1lYVtuPuou9/j7u8C7gD+tNgX7e7/4e4fKNzXga9e3LJF5qegllqQNLNU8QZ8B/iSmbWbWRvwV8C3AMzsdjNbY2YGnCTo8sib2Tozu7HwoeMkcArIh/NyROZSUEsteIwgWIu3FNAPPA+8AGwH/rZw7GXAj4Ex4GfAv7j7EwT9018BjgFDQAdw78V7CSLnZpo4QEQk2tSiFhGJOAW1iEjEKahFRCJOQS0iEnEKahGRiKvItT7a2tq8t7e3Eg8tIlKTBgYGjrl7+3z7KhLUvb299Pf3V+KhRURqkpntP9c+dX2IiERcZILa3fnFa8fZOzwWdikiIpESmaCemM7x+w/+gn978tWwSxERiZTIBHW6McFtVy7n0ecPMTGdDbscEZHIiExQA2zu62F8OsdjLwyFXYqISGREKqjf07uE3tZmtvQfCLsUEZHIiFRQmxmb+3p45rXj7B8ZD7scEZFIiFRQA3z02i5iBlsHBsMuRUQkEiIX1MsXNfGBy9p5eGCQXF7XyhYRiVxQA9zV182hk5M8/cqxsEsREQldJIP65suXsagpyUP96v4QETlvUJtZj5k9YWYvmtlOM/tspYtKJeN85JoVPL5ziJMTM5V+OhGRSFtIizoL3OPuG4DrgE+b2YbKlgV39fUwnc3zyPOHKv1UIiKRdt6gdvfD7r69sDwK7AK6Kl3Yu1dcwvrOjMZUi0jde1t91GbWC2wEnqlINXOfi819PTw/eJI9Q6OVfjoRkchacFCbWQvwMPA5d39znv13m1m/mfUfPXq0LMXdec0KknFTq1pE6tqCgtrMkgQh/W133zbfMe5+n7v3uXtfe/u8kxS8ba0tjdy0fhnf33GQmVy+LI8pIlJtFjLqw4AHgF3u/rXKlzTX5r5uRsan+cnu4Yv91CIikbCQFvX1wKeAG83sl4Xbhytc16zfWNtOe6aRLRpTLSJ16rxzJrr7TwG7CLXMKxGP8dFru7j/qdcYHp2kI5MKqxQRkVBE8puJZ9q8qYdc3vnBjoNhlyIictFVRVCv6Whh48rFbOkfxF0XahKR+lIVQQ3BNxVfHh7jucGTYZciInJRVU1Q337VclLJGA9pTLWI1JmqCepMKsmHrljOfz13iMmZXNjliIhcNFUT1BCMqR6dzPL4Tk1+KyL1o6qC+rrVrXQvaVL3h4jUlaoK6ljM+Pimbp5+ZYTBNybCLkdE5KKoqqAG+PimbgAeHtCYahGpD1UX1N1Lmnn/pa1s3X6AvCa/FZE6UHVBDcE3FQ8cP8XPXxsJuxQRkYqryqC+9YpOMqkEW3WhJhGpA1UZ1KlknN++egWP/eowo5Oa/FZEaltVBjXA5k3dTM7kefT5w2GXIiJSUVUb1Nf0LGZNR4um6RKRmle1QW1m3NXXzfbXT7B3eCzsckREKqZqgxrgzo1dxGPGlgG1qkWkdlV1UHdkUtywrp1t2w+S1eS3IlKjqjqoATb39XB0dIonXz4adikiIhVR9UF94/oOWtMNPPSsxlSLSG2q+qBOxmPcubGL/9l9hOPj02GXIyJSdlUf1BBM0zWT0+S3IlKbaiKo13VmuKp7EQ/1H9DktyJSc2oiqCH4puLuoVF2Hnoz7FJERMqqZoL6jqu7aEjE9E1FEak5NRPUi5qT3PLuTn7wS01+KyK1pWaCGoLuj5OnZvjxriNhlyIiUjY1FdTXr2lj+aIUW3SdahGpITUV1PHC5LdPvXyUwydPhV2OiEhZ1FRQQzD5bd5h23aNqRaR2nDeoDazb5rZsJn96mIU9E6tak3z3tVL2aIx1SJSIxbSov534NYK11FWd/X1sG9kgv79b4RdiojIO3beoHb3J4HjF6GWsvnwlZ2kG+IaUy0iNaHm+qgBmhsS3HbVch59/jDjU9mwyxEReUfKFtRmdreZ9ZtZ/9Gj4V8b+q6+Hiamczz2gia/FZHqVragdvf73L3P3fva29vL9bAXbNOqJbyrLc2WAY2pFpHqVpNdHxBMfvuxTd384rXj7Ds2HnY5IiIXbCHD874D/AxYZ2aDZvaHlS+rPD52bTcxg61qVYtIFVvIqI9Puvtyd0+6e7e7P3AxCiuHzkUpPri2nYe3D5LLa0y1iFSnmu36KNq8qYfDJyf56d5jYZciInJBaj6ob97QweLmpMZUi0jVqvmgbkzEufOaLn704hFOTGjyWxGpPjUf1BBcqGk6m+eR5w6FXYqIyNtWF0F9RdciLl9+ia5TLSJVqS6CGuCuvm5eOHiS3UOa/FZEqkvdBPVHrukiGTe1qkWk6tRNUC9NN3Dz5cv4/o6DTGfzYZcjIrJgdRPUEFyo6fj4ND/ZPRx2KSIiC1ZXQf3rl7XRkWlk64DGVItI9airoE7EY3z02m6e2HOU4dHJsMsREVmQugpqgM193eTyzvc1+a2IVIm6C+pL21vYtGoJWwYGNfmtiFSFugtqgM2butk7PMaOAyfCLkVE5LzqMqhvu2o5Tcm4xlSLSFWoy6DOpJJ86MpOHn3uEKemc2GXIyLyluoyqCG4TvXoVJbHdw6FXYqIyFuq26B+3+ql9Cxt4iFdp1pEIq5ugzoWMzZv6uHpV0Y4cHwi7HJERM6pboMa4GObujGDh7frQ0URia66DuquxU1cf2kbW/oHyWvyWxGJqLoOagi+qXjwxCl+/upI2KWIiMyr7oP6lnd3kkkl2DKg7g8Riaa6D+pUMs4dV6/gsRcO8+bkTNjliIicpe6DGoLrVE9l8zz63OGwSxEROYuCGriqexFrl7WwRdepFpEISoRdQBSYBWOq/+6xXewdHmVNRybskqSKuDvukHMnlz+9nHcnnw+25Z1gvXBMPh+s59xxd3J5CscVbyXr+eJxwbacOzh0LWlidVuaZFztrVqnoC64c2MXX/3hbrb0D3Lvhy8PuxwJ2cmJGfaNjLNvZJzXRybYNzLB/pFx9h+fYGwyGwRx/nSohiUZNy5tb2HtsgzrOjOsXZZhfWeGrsVNxGIWXmFSVgrqgvZMIzes72DbjoP82S3rSKiVUtPcnWNj0+wfGWffyASvF34W10+emvvBcuclKVa1NnPDunYWNzdgBnEzYmbEYlZYDr7xGo8Vlgv747HgmNiZ94mdPiY47tz3scK2eCx4B+gOB45PsOfIKC8NjTKw/w0eee7QbL3NDXEuW5Zh/bIMazszrFuWYW1nC+0tjZgpwKuNgrrE5k3d/PeLR/jfl45y0+XLwi5H3qF83hl6c/KsVnExmMdLrpwYs6Arobc1ze1XLae3Nc2q1mZ629L0LGmmqSEe4iuZ36ZVS+asj07O8PLwGHuGRtkzNMpLR0b58a4jfK/kejZL0w2sXdZSCO6g9X3ZsgyXpJIXu3x5GxYU1GZ2K/ANIA7c7+5fqWhVIblhfQdtLQ081H/ggoLa3ZmcyTM+nWViKsfYVJaJ6Szj0zkmprKF9dw5949PFfZN5xifynJqJgfFt9UljaDiYmnLqLg4774zjindavM+bnE9WIjHjHRjnOaGBC2NCZob4rQ0Jkg3JmhujNPSkKC5MUHLGcekGwvLjcHxTcl42Vtz2VyegydOzdsqfv34BNPZ/OyxybjRs7SZ3tY071u9lN7WZla1peltTdO1uImGRHW/i8qkkly7cgnXrpwb4MfGpnhpaJQ9R4IA33NklK0Dg3P+UK1YlApa3sXW97IMazpaSCWj9wcqCtyd6VyeyZk8UzM5JmfyTGZz5N1Z33lJ2Z/vvEFtZnHgn4HfBAaBZ83sEXd/sezVhCwZj/E7G7t48P/28b1nX2cqm2d8KheE6VQQnqVBWgzc0p8L7a+MGaQbE6QbTgdZc0OczktSs6HXmIgTM8M5/aDzzR5WnFLM5zmmeN+528487uzHLz1+Jp+ffX3jU1mOjU3N/tEZm8rOCcO3YkbwegtB39wYJ10M9pKgD85LEPTpkmNOzeTOahUPvnGKbMlJTyVj9LameVdbmhvXdwSt4tY0K5c2s2JxE/E67Ldta2mkbU0j71/TNrvN3Tl44tRscL80NMruoVGe3jvCdC7494wZ9LalZ4N7fWfQCl+1tDlSXYPF0JzK5pmcyTE1E/wshufstmxh20xhW7YQstnT22b3l+ybKt2XPX3f+X4X2zONPPvFm8v+Gu188waa2a8BX3b3Wwrr9xZOzt+f6z59fX3e399fzjovmr3Do/zWPz05J3BLA6YYHs0NJWHScDp0SvefDp6zW5qNiVjN9BXO5IIgH5vOznnnUHzHMDZVfMdQWJ4++5jZP4RT2TktvflkGhOsamtmVWs6aBW3pme7Kjoy6oN9J2ZyefaPjLN7aHS2Ff7SkTH2jYzPBlNDIsaa9hbWd2a4tKOFmBm5fJ5s3snmnGzeyeXzzOSCES7B9vzp5XyebGHfTOHY4v2yZ6zn8s5MyX3PXM/m8u/4w9xUMkYqGSeViJNKxmgs/kzGC9sL+0v2pQr7GhNzl1tSCW5Y13FBdZjZgLv3zbdvIV0fXUDpAONB4H0XVEkVWNOR4anP30g2l58N2Uq8Za8lyXiMRc0xFjWXp58zn3dOzeRmQ7sY4MlE0Fpe0pzUv0eFJOMx1nRkgiGqV53efmo6x97hsUL3yZvsOTLG06+MsG3HwTn3jxkkYjHiMSMRNxIxIx6LkYwHH4YmYkYiHitsn7ucSsaIx4L1ROH+8ViM5DzHJgv7EiXPk0rGg3BNzBekZ4RxYV+1NJjK9mGimd0N3A2wcuXKcj1sKLoWN4VdQl2LxazwzkSfdUdFU0OcK7sXcWX3ojnbi1PZFUNYQwIrYyEdTQeBnpL17sK2Odz9Pnfvc/e+9vb2ctUnIhHW1BCnqSFOQyKmkK6ghQT1s8BlZrbazBqATwCPVLYsEREpOu97S3fPmtlngMcJhud90913VrwyEREBFjDq44Ie1OwosP8C794GHCtjOdVM52IunY+5dD5Oq4Vzscrd5+03rkhQvxNm1n+uISr1RudiLp2PuXQ+Tqv1cxGdUesiIjIvBbWISMRFMajvC7uACNG5mEvnYy6dj9Nq+lxEro9aRETmimKLWkRESkQmqM3sVjPbY2Z7zewLYdcTJjPrMbMnzOxFM9tpZp8Nu6awmVnczHaY2aNh1xI2M1tsZlvNbLeZ7SpcOK1umdmfFH5PfmVm3zGzVNg1lVskgrrkUqofAjYAnzSzDeFWFaoscI+7bwCuAz5d5+cD4LPArrCLiIhvAD909/XA1dTxeTGzLuCPgT53v4LgS3mfCLeq8otEUAPvBfa6+6vuPg18F/hIyDWFxt0Pu/v2wvIowS9iV7hVhcfMuoHbgPvDriVsZrYI+CDwAIC7T7v7iVCLCl8CaDKzBNAMHDrP8VUnKkE936VU6zaYSplZL7AReCbkUsL0deDPgYXNUFDbVgNHgQcLXUH3m1k67KLC4u4HgX8EXgcOAyfd/UfhVlV+UQlqmYeZtQAPA59z9zfDricMZnY7MOzuA2HXEhEJ4FrgX919IzAO1O1nOma2hODd92pgBZA2s98Lt6ryi0pQL+hSqvXEzJIEIf1td98Wdj0huh64w8z2EXSJ3Whm3wq3pFANAoPuXnyHtZUguOvVzcBr7n7U3WeAbcD7Q66p7KIS1LqUagkLppx4ANjl7l8Lu54wufu97t7t7r0E/y9+4u4112JaKHcfAg6Y2brCppuAmpu/9G14HbjOzJoLvzc3UYMfrkZiCg1dSvUs1wOfAl4ws18Wtv2Fuz8WXkkSIX8EfLvQqHkV+IOQ6wmNuz9jZluB7QSjpXZQg99S1DcTRUQiLipdHyIicg4KahGRiFNQi4hEnIJaRCTiFNQiIhGnoBYRiTgFtYhIxCmoRUQi7v8Bzzz2ptI0yuAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAACSCAYAAABLwAHLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAYGUlEQVR4nO3de3CV933n8fcX3e8XECBAIGzAlxjb2KpvbJqrE99ip5t27SROINvWMztxk/SetJ2k43ZnMt2dbbPTbGtPaoNT12zjuBnqgJ00sZOMsQlgWOO7CQgkIYFA6IZ0JB3pu388j+BICOsARzxHz/m8ZjTPeW7nfPXA+fye5/f8dI65OyIiEl9zoi5ARERmloJeRCTmFPQiIjGnoBcRiTkFvYhIzCnoRURiTkEvIhJzCnqJFTN7wcxOmFlR1LWIZAsFvcSGmTUC7wccuPsivm7+xXotkfOhoJc4+TzwMrABWDe+0MwazOxpM+s0s+Nm9vcp637XzN40sz4ze8PMrguXu5mtSNlug5n9dfj4g2bWamZ/amYdwGNmVmNmz4SvcSJ8vCRl/1oze8zMDofrfxAuf83MPpGyXYGZHTOzNTN1kCT3KOglTj4PPBH+fNzMFphZHvAMcBBoBBYDmwDM7LeAvwz3qyS4Cjie5mstBGqBZcADBO+lx8L5pcAg8Pcp238XKAXeB8wH/jZc/jhwf8p2dwDt7r47zTpEpmX6rBuJAzP7T8DzQL27HzOzt4CHCc7wN4fLk5P2eQ7Y4u7fmuL5HFjp7vvC+Q1Aq7v/hZl9EPgRUOnuibPUcy3wvLvXmFk90AbMdfcTk7ZbBLwNLHb3XjN7Cvilu//NeR4KkTPojF7iYh3wI3c/Fs7/S7isATg4OeRDDcCvzvP1OlND3sxKzexhMztoZr3Az4Hq8IqiAeiaHPIA7n4YeBH4lJlVA7cTXJGIZIxuIsmsZ2YlwH8B8sI+c4AioBo4Aiw1s/wpwr4FuPQsTztA0NUybiHQmjI/+VL4D4HLgBvdvSM8o98NWPg6tWZW7e7dU7zWRuB3CN6PL7l721lqEjkvOqOXOPgkMApcCVwb/lwB/CJc1w5808zKzKzYzNaG+30H+CMzu94CK8xsWbhuD/AZM8szs9uAD0xTQwVBv3y3mdUC3xhf4e7twFbg/4Q3bQvM7NdT9v0BcB3wZYI+e5GMUtBLHKwDHnP3Q+7eMf5DcDP008AngBXAIYKz8nsB3P17wH8n6ObpIwjc2vA5vxzu1w18Nlz3Xv4OKAGOEdwXeHbS+s8BI8BbwFHgK+Mr3H0Q+D6wHHg6/V9bJD26GSuSBczs68Aqd79/2o1FzpH66EUiFnb1/DbBWb9IxqnrRiRCZva7BDdrt7r7z6OuR+JJXTciIjGnM3oRkZhT0IuIxFzW3YydN2+eNzY2Rl2GiMissmvXrmPuXjfVummD3sweBe4Cjrr7VVOsN+BbBB/GNACsd/dXwnXrgL8IN/1rd9843es1Njayc+fO6TYTEZEUZnbwbOvS6brZANz2HutvB1aGPw8A/xC+6PhfB94I3AB8w8xq0itZREQyZdozenf/efiFDmdzD/C4B8N3Xjaz6vDT+j4I/NjduwDM7McEDcaTF1y1TOt4/xBvdfQxt7yQhZXFVJUUEFx85ZaB4SQdPQk6eoPPH6stK6S2tJDq0kIK83WLKheNjjn9iSS9iRF6BkfoTYwwnByjojif8qICyovzKS8KfvLmxOM9k4k++sUE44DHtYbLzrb8DGb2AMHVAEuXLs1ASbnpWP8Qz77WwZa97by8/zhjKSNni/LnsLCqmAWVxSysLGZBZVHwuGp8PviZLeE3OuYc6x/iSG+Cjp5EMO1N0NEzdOrxkZ4EfUNTfWhloKIon5qyQmrKCqktLQinhdSWB9OaskJqywqpKQ2mVSUFsXnjz2ZTBXXvYDKcjtCbSIbTicv7wuXv9X9istLCvLAByKe8uICKovHHwfT0umBaWTyxoagozqesKJ+CvGjfV1lxM9bdHwEeAWhqatLA/nPQ2TfEs693sOXVdrYfCML9kroyvvihFdy4fC7dg8MpQRiE4P9r7aajJ8FQcuyM55tbVniqATjVGFQWsyBsEBZWFlNdOrNXB/1DwVn40fHwDkO7Y/x36EnQ2T/E6NjE/yp5c4z5FUHNK+rKWXvp3Al1Y3Di5AhdA8OcODlM18lhTgwE087+Id450k/XyWEGR0anrMsMqktSGoTxhqAstWEoONUw1JQVUlGUn5NXUtNxd7oHRjjcM5hWSPecY1BXFAehW1lSQGVxPg21peH8xOXBtIDCfKN/aJT+RJL+oeC1+oeS4XySvqFksCwxwtG+BP2JYFn/UJJ0/hSpuGAO5UUFpxuG8UaiOD9oPMKriYbaEu66elEGjvBEmQj6NoLP2x63JFzWRtB9k7r8hQy8Xs472pfgudc6+OHedn55oIsxh0vrynjwQyu44+p6LltQMW24uDs9gyPhWXDYEPQM0dF7OmBfbe3mWP/wGfsW5c8JrwBONwSnrhbCYJ1fWURRft6E/UbHnM6+oVOvebQvcapb5fSZ+RD9U7yRK4rzT73OyvnzprwqmVtelJEz7sHh0VMNwPi062TYOAwMB43FyWEOdQ2wp6WbEwPDjIxO/W4vyLPTwR9O6yqKWFRdzMKqEhZVnT52UZ/1ZVpvYoTWrkFaTgzQemKQlq5g2hrOT/XvPK6iKAjhijCMl9SUnhHSVSUFp4I6dd3F7HJxdwaGR4PGIJGkLzFyqoHoS2koxtcH64KG5FDXwOllQ0lGx5zrl9XMSNCn9ZexYR/9M2cZdXMn8CDBqJsbgf/t7jeEN2N3EXz8KsArwPXjffZn09TU5Bp1c6ajvQm2huG+o7kLd1gxv5w7Vtdz5+p6Vi0on5Ezx+HkGEf7JjYEqd0l490kiZEzrw5qSgtYUFlMUf4cOnoTdPYNMekknPzxs/CULqTUxmI83EsLs+Lic0ruTv9QcsqrheMnz5w/OkVjZgZ15UXUV5dQX1lMfXUx9VXF1FeVBNPqEuZXFGVVYzAwnDwV3C1dKdPuYNozODJh+9LCPBpqSmmoLWFJTSlLakpYVF1CdWkQ1lVhaJcXx6dvPF3uTmJkjOHkGFWlBef1HGa2y92bplqXzvDKJwnOzOeZWSvBSJqCsLh/BLYQhPw+guGVXwjXdZnZXwE7wqd6aLqQl4mO9CbYuredLXs72HEwCPeV88v50odXcufV9axaUDHjNRTmzwnflKVn3cbd6R1MntHNMt4QDCXHWLWgYsI9goVVQZDPKytizix/U5sZFcUFVBQXsHTu2Y9Tqr7ECO09CQ53D9LRk+BwT4KOnkHaexLs6+znF+92cnJ4YheSGcyvKJpwJbCoqiSYhlcImWwMEiOjHO4epGVymJ8YpLVrgOMnJ17tFeXPYUlNEOLXNlSzpKZ0QrDXzHCX32xmZpQU5lFSmDf9xufz/Nn2WTe5fkbf0ZNg62vtbNnbzs6DJ3CHVQtOn7mvvAjhLtFzd/qGkrR3J2gPG4D2ngTt3YN09AYNRHtPgoFJjcEcg7ppGoMFFUXk581hZHSM9u5EGN4Tu1daTgxwpHdownMX5BmLqkvOOCtfEs7HodGezS7ojF5mXkdPgi17T4c7wGULKvjKR1Zx59ULWTFf4Z5rzCzoc15YwGULp/73d3d6E8nwiiC4MmjvPt0ovHOkj5+90zllY1BdWkj3wPCErrQ5BvVVJTTUlvD+lXU0hEHeUBtMF1QW51yXSlwo6CPS3jPIlr3BUMhdYbhfvrCCP7h1FXesrmfF/PKIK5RsZ2ZUlQR929M1BqeuCrqDLqLO/iHmlRdNCPOFVfG7ISwBBf1FdLh78NSZ+yuHugG4or6SP/rYKm5fXc+ldQp3yazUxuDyhZVRlyMRUdDPsLbuQbbubeeHe9vZHYb7lfWV/PHHL+P2qxZyicJdRGaYgn4GtJ4YYOveYCjknpZuAN63KAj3O1bXs3xeWbQFikhOUdBn2K6DXdz78Mskx5yrFlfyJ7ddxh1X1dOocBeRiCjoM+wff7afiuJ8fvDFtSybq3AXkejpFnsGtXQN8B9vHuEzNy5VyItI1lDQZ9DjLzUzx4z7b1oWdSkiIqco6DPk5FCSTTtauO2qhdRXlURdjojIKQr6DPm33W30JZL817WNUZciIjKBgj4D3J0N25pZvbiK65bq2xJFJLso6DPgxX3H2Xe0n/W3NOrT+UQk6yjoM2DDtgPMKy/krmvqoy5FROQMCvoLdPD4SX7y1lE+c8PSM75RSUQkGyjoL9DjLx0kz4zPakiliGSptILezG4zs7fNbJ+ZfXWK9X9rZnvCn3fMrDtl3WjKus0ZrD1y/UNJ/nVHC3esrmdBZXHU5YiITCmdrxLMA74N3Aq0AjvMbLO7vzG+jbv/fsr2vwesSXmKQXe/NmMVZ5GnX2mlbyjJeg2pFJEsls4Z/Q3APnff7+7DwCbgnvfY/tPAk5koLpuNjQVDKq9ZUsWahuqoyxEROat0gn4x0JIy3xouO4OZLQOWAz9NWVxsZjvN7GUz++T5FpptfrHvGPs7T7J+rYZUikh2y/SnV94HPOXuqV9Suczd28zsEuCnZrbX3X+VupOZPQA8ALB06dIMlzQzNrx4gLqKIu5cvSjqUkRE3lM6Z/RtQEPK/JJw2VTuY1K3jbu3hdP9wAtM7L8f3+YRd29y96a6uro0SorWgWMnef7tTj5741IK8zVwSUSyWzoptQNYaWbLzayQIMzPGD1jZpcDNcBLKctqzKwofDwPWAu8MXnf2WbjtmYK8ozP3Dg7rj5EJLdN23Xj7kkzexB4DsgDHnX3183sIWCnu4+H/n3AJnf3lN2vAB42szGCRuWbqaN1ZqO+xAhP7WrlrqsXMb9CQypFJPul1Ufv7luALZOWfX3S/F9Osd82YPUF1Jd1vr+rlf6hJOtvaYy6FBGRtKiD+RyMjTkbXzrImqXVXKMhlSIySyjoz8HP3unkwLGTOpsXkVlFQX8OHtvWzPyKIm6/Sp9SKSKzh4I+TfuO9vPzdzq5/6ZlGlIpIrOKEitNj7/UTGHeHD59g4ZUisjsoqBPQ284pPIT1yyirqIo6nJERM6Jgj4N39vZysDwqG7CisispKCfxuiYs3FbM03Lali9pCrqckREzpmCfhovvH2UQ10D+sx5EZm1FPTT2LCtmYWVxXz8fQujLkVE5Lwo6N/Du0f6+MW7x/jczcsoyNOhEpHZSen1HjZsa6Ywfw73/VrD9BuLiGQpBf1Z9AyM8PQrbdxzzSLmlmtIpYjMXgr6s/jXnS0MjozqJqyIzHoK+imMjjkbX2rmhuW1vG+RhlSKyOymoJ/CT948QuuJQb6gP5ASkRhQ0E9hw7ZmFlUVc+uVC6IuRUTkgqUV9GZ2m5m9bWb7zOyrU6xfb2adZrYn/PmdlHXrzOzd8GddJoufCW939LHtV8f53M2N5GtIpYjEwLRfJWhmecC3gVuBVmCHmW2e4rtf/6+7Pzhp31rgG0AT4MCucN8TGal+BmzY1kyRhlSKSIykc8p6A7DP3fe7+zCwCbgnzef/OPBjd+8Kw/3HwG3nV+rM6x4Y5t92t/IbaxZTU1YYdTkiIhmRTtAvBlpS5lvDZZN9ysxeNbOnzGz8dDjdfbPCph0tJEbGWKebsCISI5nqhP53oNHdryY4a994Ljub2QNmttPMdnZ2dmaopHOTHB3juy8d5KZLarmivjKSGkREZkI6Qd8GpHZYLwmXneLux919KJz9DnB9uvuG+z/i7k3u3lRXV5du7Rn1H28eoa17kPW3LI/k9UVEZko6Qb8DWGlmy82sELgP2Jy6gZmlflv23cCb4ePngI+ZWY2Z1QAfC5dlncdebGZxdYmGVIpI7Ew76sbdk2b2IEFA5wGPuvvrZvYQsNPdNwNfMrO7gSTQBawP9+0ys78iaCwAHnL3rhn4PS7IG4d72X6giz+743Ly5ljU5YiIZNS0QQ/g7luALZOWfT3l8deAr51l30eBRy+gxhm3cVszJQV53NukL/4WkfjJ+b8I6jo5zA/2tPEb1y2mqrQg6nJERDIu54N+045DDCXH9MXfIhJbOR3040Mq166Yy6oFFVGXIyIyI3I66J97/QjtPQkNqRSRWMvpoN+w7QANtSV8+PL5UZciIjJjcjboX2vrYUfzCdbd3KghlSISazkb9BvCIZW/1aRPqRSReMvJoD/WP8TmPYf51PWLqSrRkEoRibecDPpNvzzE8KiGVIpIbsi5oB8ZHeO7Lx/k/SvnsWK+hlSKSPzlXNA/+1oHR3qH+MLaxqhLERG5KHIu6Ddsa2bZ3FI+uEpDKkUkN+RU0L/a2s2ug8GQyjkaUikiOSKngn7Di82UFebxm01Loi5FROSiyZmgP9qX4N9fPcxvXr+EymINqRSR3JEzQf/k9hZGRp3Pa0iliOSYnAj64eQY/7z9IB9YVceldeVRlyMiclGlFfRmdpuZvW1m+8zsq1Os/wMze8PMXjWzn5jZspR1o2a2J/zZPHnfi2Hra+109g2xXkMqRSQHTftVgmaWB3wbuBVoBXaY2WZ3fyNls91Ak7sPmNl/A/4GuDdcN+ju12a27HPz2IvNXDKvjA+srIuyDBGRSKRzRn8DsM/d97v7MLAJuCd1A3d/3t0HwtmXgawZ1rL70An2tHSz7hYNqRSR3JRO0C8GWlLmW8NlZ/PbwNaU+WIz22lmL5vZJ6fawcweCLfZ2dnZmUZJ6du4rZnyonw+dX3WtD0iIhfVtF0358LM7geagA+kLF7m7m1mdgnwUzPb6+6/St3P3R8BHgFoamryTNVztDfBD/e2c/9NyygvyuivKiIya6RzRt8GpH5o+5Jw2QRm9lHgz4G73X1ofLm7t4XT/cALwJoLqPecPLH9EMkxZ93NjRfrJUVEsk46Qb8DWGlmy82sELgPmDB6xszWAA8ThPzRlOU1ZlYUPp4HrAVSb+LOmKHkKE9sP8iHLptP47yyi/GSIiJZadr+DHdPmtmDwHNAHvCou79uZg8BO919M/A/gHLge2YGcMjd7wauAB42szGCRuWbk0brzJgfvtrOsf5hfea8iOS8tDqu3X0LsGXSsq+nPP7oWfbbBqy+kALPh7vz2IvNXFpXxvtXzrvYLy8iklVi+ZexrxzqZm9bD+tvaSS8whARyVmxDPoN25qpKM7nP1+nIZUiIrEL+o6eBFv3tnNvUwNlGlIpIhK/oH9i+0FG3fm8hlSKiAAxC/rEyCj/sv0QH7l8AUvnlkZdjohIVohV0D/zajvHTw7ri79FRFLEJuiDIZUHWDm/nFsunRt1OSIiWSM2QX/w+ADvHuln/VoNqRQRSRWbYSmN88rY9rUPU1YYm19JRCQjYpWK88qLoi5BRCTrxKbrRkREpqagFxGJOXPP2Pd8ZISZdQIHL+Ap5gHHMlTObKdjMZGOx0Q6HqfF4Vgsc/cpvxg764L+QpnZTndvirqObKBjMZGOx0Q6HqfF/Vio60ZEJOYU9CIiMRfHoH8k6gKyiI7FRDoeE+l4nBbrYxG7PnoREZkojmf0IiKSIjZBb2a3mdnbZrbPzL4adT1RMrMGM3vezN4ws9fN7MtR1xQ1M8szs91m9kzUtUTNzKrN7Ckze8vM3jSzm6OuKUpm9vvh++Q1M3vSzIqjrinTYhH0ZpYHfBu4HbgS+LSZXRltVZFKAn/o7lcCNwFfzPHjAfBl4M2oi8gS3wKedffLgWvI4eNiZouBLwFN7n4VkAfcF21VmReLoAduAPa5+353HwY2AfdEXFNk3L3d3V8JH/cRvJEXR1tVdMxsCXAn8J2oa4mamVUBvw78E4C7D7t7d6RFRS8fKDGzfKAUOBxxPRkXl6BfDLSkzLeSw8GWyswagTXA9ohLidLfAX8CjEVcRzZYDnQCj4VdWd8xs7Koi4qKu7cB/xM4BLQDPe7+o2iryry4BL1MwczKge8DX3H33qjriYKZ3QUcdfddUdeSJfKB64B/cPc1wEkgZ+9pmVkNwdX/cmARUGZm90dbVebFJejbgIaU+SXhspxlZgUEIf+Euz8ddT0RWgvcbWbNBF16Hzazf462pEi1Aq3uPn6F9xRB8OeqjwIH3L3T3UeAp4FbIq4p4+IS9DuAlWa23MwKCW6mbI64pshY8BVb/wS86e7/K+p6ouTuX3P3Je7eSPD/4qfuHrsztnS5ewfQYmaXhYs+ArwRYUlROwTcZGal4fvmI8Tw5nQsvnjE3ZNm9iDwHMFd80fd/fWIy4rSWuBzwF4z2xMu+zN33xJdSZJFfg94Ijwp2g98IeJ6IuPu283sKeAVgtFqu4nhX8nqL2NFRGIuLl03IiJyFgp6EZGYU9CLiMScgl5EJOYU9CIiMaegFxGJOQW9iEjMKehFRGLu/wNC+jmbHxTDGgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#调用\n",
    "epoch = 5\n",
    "Loss = []\n",
    "Accuracy = []\n",
    "for epoch in range(1, epoch+1):\n",
    "    print(\"start_time\",time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())))\n",
    "    loss, acc = train_runner(model, device, trainloader, optimizer, epoch)\n",
    "    Loss.append(loss)\n",
    "    Accuracy.append(acc)\n",
    "    test_runner(model, device, testloader)\n",
    "    print(\"end_time: \",time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())),'\\n')\n",
    "\n",
    "print('Finished Training')\n",
    "plt.subplot(2,1,1)\n",
    "plt.plot(Loss)\n",
    "plt.title('Loss')\n",
    "plt.show()\n",
    "plt.subplot(2,1,2)\n",
    "plt.plot(Accuracy)\n",
    "plt.title('Accuracy')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7517214-c63d-4f3e-a6f1-e6dea60658d8",
   "metadata": {},
   "source": [
    "## 1.7 保存模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e6a93bc8-ec65-45f2-9f07-b9b225c77801",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LeNet(\n",
      "  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (relu): ReLU()\n",
      "  (maxpool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (maxpool2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (fc1): Linear(in_features=400, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Administrator\\.conda\\envs\\pytorch\\lib\\site-packages\\torch\\serialization.py:360: UserWarning: Couldn't retrieve source code for container of type LeNet. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n"
     ]
    }
   ],
   "source": [
    "print(model)\n",
    "torch.save(model, './models/model-mnist.pth') #保存模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b79d9d23-feb1-43b5-81b7-16e41907d153",
   "metadata": {},
   "source": [
    "## 1.8 手写图片的测试"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "474173d9-de04-40e0-8904-5480f186387d",
   "metadata": {},
   "source": [
    "利用刚刚训练的模型进行手写图片的测试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "18668036-1fdf-4988-93d5-942770007ee6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAHj0lEQVR4nO3dzYuVdR/H8XNGU9MyEVScRUYPBj0gBMO4CdvIYCVMA4NS+Be0SOgfqE0LQ8hFSkFS6E4ighYJgxihEFGgGEQ1bdQhxJGpKJB5utd39/W9bseH/BzP67W8Plx1sN4eOD/OubqLi4sdIM/A3X4BQDNxQihxQihxQihxQqjlbWO32/VRLtxhi4uL3abr3jkhlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghlDghVOvjGO5VjzzySLlNTEyU24ULF8ptdHT0Fl4R/C/vnBBKnBBKnBBKnBBKnBBKnBCqL49SRkZGyu2HH34ot9nZ2TvxcqCRd04IJU4IJU4IJU4IJU4IJU4I1ZdHKYODg+W2ffv2cnvnnXfuxMuBRt45IZQ4IZQ4IZQ4IZQ4IZQ4IVRfHqWcOHGi3MbHx8vtp59+uhMvBxp554RQ4oRQ4oRQ4oRQ4oRQfflp7fr168vt6tWr5fbpp5+W2+rVq8ttYKD+O3BxcXFJ11mabrdbbul/xt45IZQ4IZQ4IZQ4IZQ4IZQ4IVRfHqU8+OCD5TY3N1duCwsL5fbee++V2xNPPFFup06darw+NTVV3tP2ZO4VK1aU28mTJ8utcuXKlXJbtmxZue3du7fc5ufny+3xxx9vvD40NFTe89RTT5Vb22tM550TQokTQokTQokTQokTQokTQvXlUcrly5fL7aGHHiq3l19+udyOHDlSbpOTk+X27rvvNl7/+eefy3u2bt1abm32799fbuvWrVvyP++PP/4ot7Zv4rR9g6f6s2r7Bknb8Vcv884JocQJocQJocQJocQJocQJofryKOXpp58utwceeKDcDh06VG7vv/9+ubUdU3zzzTeN19u+DdL2rZQnn3yy3Kanp8tt7dq1jdfbHk/x448/ltvMzEy5tR3BVN8i+eqrr8p7vvvuu3LzA1/AbSdOCCVOCCVOCCVOCCVOCNVt+zi52+1mf9Z8k+6///5yazvCaDs6OHDgQLm1PUkbFhcXG897vHNCKHFCKHFCKHFCKHFCKHFCqL48Smn7psJrr71WbseOHSu36tslnU6ns3379ht7YfQlRynQY8QJocQJocQJocQJofry09o2y5fXP6vU9rs4Z86cKbeRkZFbeUnc43xaCz1GnBBKnBBKnBBKnBBKnBCqLx/H0Oa+++4rt7Zjp7ajlNvtZh8x0HZMVLlXnxrdC7xzQihxQihxQihxQihxQihxQijfSvmHFStWlFvbE5nb7nvhhRfKrXpUQ/WE506n07l27Vq5/fbbb+X2/PPPl9vU1FTj9Y8//ri8Z9++feX2ySeflNvbb79dbvPz843X059CfSt8KwV6jDghlDghlDghlDghlDghlKOUfxgYqP+++vXXX8tt06ZNN3XfuXPnGq+vXLmyvOeZZ54pt4mJiXIbHh4ut40bNzZe37x5c3nP2bNny63t/6s///yz3A4ePNh4/fTp0+U9vc5RCvQYcUIocUIocUIocUIocUIoRym3ydDQULm1/Rnv2LGj8fq3335b3nPp0qVyazu22bBhQ7n9/vvvjddnZ2fLe1599dVyO378eLkdOXKk3MbGxhqvv/jii+U933//fbn1Akcp0GPECaHECaHECaHECaF8Wsu/ru2T3Oeee67x+i+//FLeMzo6Wm4LCws3/LruFp/WQo8RJ4QSJ4QSJ4QSJ4QSJ4TyZGv+dW+88Ua5ffnll43X24781qxZU25tv1eUzjsnhBInhBInhBInhBInhBInhHKUsgQPP/xwuVW/wfP/tn40MzNTbtu2bWu83vZU8b///vtWX1Ik75wQSpwQSpwQSpwQSpwQSpwQyg98LcHnn39ebjt37iy3VatWldsrr7yy5H9XL9i1a1e5HT58uNzOnz/feP3RRx8t73n22Wdv/IUF8gNf0GPECaHECaHECaHECaHECaEcpSxBt9v4iXen0+l0Pvroo3L7+uuvy+3o0aON169evVres3v37nLbsmVLuX3xxRfl9tdff5Vbpe2456WXXiq3ZcuWldv09HTj9ccee6y8p9e/9eMoBXqMOCGUOCGUOCGUOCGUT2tvk8HBwXJr+3SyevzA5s2by3smJydv6nV89tln5Vb9ds/w8HB5z/Ll9U9QtT0G4a233iq3Dz/8sPH6zXya3Ct8Wgs9RpwQSpwQSpwQSpwQSpwQylHKXVYdR7z++uvlPRcvXiy3sbGxctuzZ8+SX8f169fLe958881y++CDD8ptbm6u3PqRoxToMeKEUOKEUOKEUOKEUOKEUI5SelDbbxm1/fccGKj/Ll6/fn3j9fHx8fKe6vePOp32Ixj+m6MU6DHihFDihFDihFDihFDihFCOUuAuc5QCPUacEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEEqcEKr1ydbA3eOdE0KJE0KJE0KJE0KJE0KJE0L9B+oztu2DpJSGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "概率： tensor([[2.0888e-07, 1.1599e-07, 6.1852e-05, 1.5797e-04, 1.4975e-09, 9.9977e-01,\n",
      "         1.9271e-06, 3.1589e-06, 1.2186e-07, 4.3405e-07]],\n",
      "       grad_fn=<SoftmaxBackward>)\n",
      "预测类别： 5\n"
     ]
    }
   ],
   "source": [
    "import cv2\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "    model = torch.load('./models/model-mnist.pth') #加载模型\n",
    "    model = model.to(device)\n",
    "    model.eval()    #把模型转为test模式\n",
    "    \n",
    "    #读取要预测的图片\n",
    "    img = cv2.imread(\"./images/test_mnist.jpg\")\n",
    "    img=cv2.resize(img,dsize=(32,32),interpolation=cv2.INTER_NEAREST)\n",
    "    plt.imshow(img,cmap=\"gray\") # 显示图片\n",
    "    plt.axis('off') # 不显示坐标轴\n",
    "    plt.show()\n",
    "    \n",
    "    # 导入图片，图片扩展后为[1，1，32，32]\n",
    "    trans = transforms.Compose(\n",
    "        [\n",
    "            transforms.ToTensor(),\n",
    "            transforms.Normalize((0.1307,), (0.3081,))\n",
    "        ])\n",
    "    img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)#图片转为灰度图，因为mnist数据集都是灰度图\n",
    "    img = trans(img)\n",
    "    img = img.to(device)\n",
    "    img = img.unsqueeze(0)  #图片扩展多一维,因为输入到保存的模型中是4维的[batch_size,通道,长，宽]，而普通图片只有三维，[通道,长，宽]\n",
    "    \n",
    "    # 预测 \n",
    "    #output = model(img)\n",
    "    #predict = output.argmax(dim=1)\n",
    "    #print(predict.item())\n",
    "    \n",
    "    \n",
    "    \n",
    "    # 预测 \n",
    "    output = model(img)\n",
    "    prob = F.softmax(output,dim=1) #prob是10个分类的概率\n",
    "    print(\"概率：\",prob)\n",
    "    value, predicted = torch.max(output.data, 1)\n",
    "    predict = output.argmax(dim=1)\n",
    "    print(\"预测类别：\",predict.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e81d545-b462-4a27-be38-d6eac794d657",
   "metadata": {},
   "source": [
    "# 二、使用LeNet-5网络结构创建CIFAR-10手写数字识别分类器"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccab3503-491b-498a-a493-63218ec2faab",
   "metadata": {},
   "source": [
    "LeNet-5网络本是用来识别MNIST数据集的，下面我们来将LeNet-5应用到一个比较复杂的例子，识别CIFAR-10数据集。\n",
    "\n",
    "CIFAR-10 是由 Hinton 的学生 Alex Krizhevsky 和 Ilya Sutskever 整理的一个用于识别普适物体的小型数据集。一共包含 10 个类别的 RGB 彩色图 片：飞机（ airlane ）、汽车（ automobile ）、鸟类（ bird ）、猫（ cat ）、鹿（ deer ）、狗（ dog ）、蛙类（ frog ）、马（ horse ）、船（ ship ）和卡车（ truck ）。图片的尺寸为 32×32 ，数据集中一共有 50000 张训练圄片和 10000 张测试图片。 CIFAR-10 的图片样例如图所示。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0db4e8fd-d3fa-49e1-937b-12a41326bc51",
   "metadata": {},
   "source": [
    "![](./images/cifar-10.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f98be375-0b9f-48c0-8a5b-545ca705b681",
   "metadata": {},
   "source": [
    "## 2.1 下载并加载数据，并做出一定的预先处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "dc488d07-207f-4951-a178-1fa6c23df3e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "pipline_train = transforms.Compose([\n",
    "    #随机旋转图片\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    #将图片尺寸resize到32x32\n",
    "    transforms.Resize((32,32)),\n",
    "    #将图片转化为Tensor格式\n",
    "    transforms.ToTensor(),\n",
    "    #正则化(当模型出现过拟合的情况时，用来降低模型的复杂度)\n",
    "    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n",
    "])\n",
    "pipline_test = transforms.Compose([\n",
    "    #将图片尺寸resize到32x32\n",
    "    transforms.Resize((32,32)),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n",
    "])\n",
    "#下载数据集\n",
    "train_set = datasets.CIFAR10(root=\"./data/CIFAR10\", train=True, download=True, transform=pipline_train)\n",
    "test_set = datasets.CIFAR10(root=\"./data/CIFAR10\", train=False, download=True, transform=pipline_test)\n",
    "#加载数据集\n",
    "trainloader = torch.utils.data.DataLoader(train_set, batch_size=64, shuffle=True)\n",
    "testloader = torch.utils.data.DataLoader(test_set, batch_size=32, shuffle=False)\n",
    "# 类别信息也是需要我们给定的\n",
    "classes = ('plane', 'car', 'bird', 'cat','deer', 'dog', 'frog', 'horse', 'ship', 'truck')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a8c515f-8810-4adb-8559-1b53e3e17059",
   "metadata": {},
   "source": [
    "## 2.2 搭建LeNet-5神经网络结构，并定义前向传播的过程"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bbf4dea-e14e-4616-a8ff-700671b1d55d",
   "metadata": {},
   "source": [
    "LeNet-5网络上文已经搭建过了，由于CIFAR10数据集图像是RGB三通道的，因此LeNet-5网络C1层卷积选择的滤波器需要3通道，网络其它结构跟上文都是一样的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "040dde60-eb67-437a-a1a6-e7bc0d0d9055",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LeNetRGB(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(LeNetRGB, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5)   # 3表示输入是3通道\n",
    "        self.relu = nn.ReLU()\n",
    "        self.maxpool1 = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        self.maxpool2 = nn.MaxPool2d(2, 2)\n",
    " \n",
    "        self.fc1 = nn.Linear(16*5*5, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    " \n",
    "    def forward(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.maxpool1(x)\n",
    "        x = self.conv2(x)\n",
    "        x = self.maxpool2(x)\n",
    "        x = x.view(-1, 16*5*5)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        output = F.log_softmax(x, dim=1)\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ea25246-4021-4266-96a1-7b7b286e60aa",
   "metadata": {},
   "source": [
    "## 2.3 将定义好的网络结构搭载到GPU/CPU，并定义优化器"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5db53e7c-fe1e-456a-a6fe-de57221bf922",
   "metadata": {},
   "source": [
    "使用SGD（随机梯度下降）优化，学习率为0.001，动量为0.9"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "cd80d05f-021e-4bd0-962d-625bf462c72b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#创建模型，部署gpu\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "model = LeNetRGB().to(device)\n",
    "#定义优化器\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)\n",
    "#optimizer = optim.Adam(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35bee4bb-d77e-4542-b9b6-2a6379309c6e",
   "metadata": {},
   "source": [
    "训练和测试函数的定义跟LeNet-5章节定义的一样。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c286804-2cd8-4e00-940e-e4bf17640c07",
   "metadata": {},
   "source": [
    "## 2.4 定义训练过程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6fd79683-21e7-4503-bf60-2ac8c2a28ddb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_runner(model, device, trainloader, optimizer, epoch):\n",
    "    #训练模型, 启用 BatchNormalization 和 Dropout, 将BatchNormalization和Dropout置为True\n",
    "    model.train()\n",
    "    total = 0\n",
    "    correct =0.0\n",
    "    \n",
    "    #enumerate迭代已加载的数据集,同时获取数据和数据下标\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        inputs, labels = data\n",
    "        #把模型部署到device上\n",
    "        inputs, labels = inputs.to(device), labels.to(device)\n",
    "        #初始化梯度\n",
    "        optimizer.zero_grad()\n",
    "        #保存训练结果\n",
    "        outputs = model(inputs)\n",
    "        #计算损失和\n",
    "        #多分类情况通常使用cross_entropy(交叉熵损失函数), 而对于二分类问题, 通常使用sigmod\n",
    "        loss = F.cross_entropy(outputs, labels)\n",
    "        #获取最大概率的预测结果\n",
    "        #dim=1表示返回每一行的最大值对应的列下标\n",
    "        predict = outputs.argmax(dim=1)\n",
    "        total += labels.size(0)\n",
    "        correct += (predict == labels).sum().item()\n",
    "        #反向传播\n",
    "        loss.backward()\n",
    "        #更新参数\n",
    "        optimizer.step()\n",
    "        if i % 1000 == 0:\n",
    "            #loss.item()表示当前loss的数值\n",
    "            print(\"Train Epoch{} \\t Loss: {:.6f}, accuracy: {:.6f}%\".format(epoch, loss.item(), 100*(correct/total)))\n",
    "            Loss.append(loss.item())\n",
    "            Accuracy.append(correct/total)\n",
    "    return loss.item(), correct/total"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3e6b611-390a-409d-b74c-bca04b28aaa9",
   "metadata": {},
   "source": [
    "## 2.5 定义测试过程"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bc964210-8248-4573-ac24-c692a33d1389",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_runner(model, device, testloader):\n",
    "    #模型验证, 必须要写, 否则只要有输入数据, 即使不训练, 它也会改变权值\n",
    "    #因为调用eval()将不启用 BatchNormalization 和 Dropout, BatchNormalization和Dropout置为False\n",
    "    model.eval()\n",
    "    #统计模型正确率, 设置初始值\n",
    "    correct = 0.0\n",
    "    test_loss = 0.0\n",
    "    total = 0\n",
    "    #torch.no_grad将不会计算梯度, 也不会进行反向传播\n",
    "    with torch.no_grad():\n",
    "        for data, label in testloader:\n",
    "            data, label = data.to(device), label.to(device)\n",
    "            output = model(data)\n",
    "            test_loss += F.cross_entropy(output, label).item()\n",
    "            predict = output.argmax(dim=1)\n",
    "            #计算正确数量\n",
    "            total += label.size(0)\n",
    "            correct += (predict == label).sum().item()\n",
    "        #计算损失值\n",
    "        print(\"test_avarage_loss: {:.6f}, accuracy: {:.6f}%\".format(test_loss/total, 100*(correct/total)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "601d32c5-7719-4305-8564-d7d2b91f70c6",
   "metadata": {},
   "source": [
    "## 2.6 运行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e5c60ef5-bffc-48ac-a09b-6ec63d8b4882",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start_time 2021-11-27 22:18:10\n",
      "Train Epoch1 \t Loss: 2.309664, accuracy: 14.062500%\n",
      "test_avarage_loss: 0.045601, accuracy: 46.390000%\n",
      "end_time:  2021-11-27 22:18:44 \n",
      "\n",
      "start_time 2021-11-27 22:18:44\n",
      "Train Epoch2 \t Loss: 1.271340, accuracy: 57.812500%\n",
      "test_avarage_loss: 0.039428, accuracy: 55.340000%\n",
      "end_time:  2021-11-27 22:19:19 \n",
      "\n",
      "start_time 2021-11-27 22:19:19\n",
      "Train Epoch3 \t Loss: 1.245964, accuracy: 57.812500%\n",
      "test_avarage_loss: 0.037095, accuracy: 58.200000%\n",
      "end_time:  2021-11-27 22:19:54 \n",
      "\n",
      "start_time 2021-11-27 22:19:54\n",
      "Train Epoch4 \t Loss: 0.966007, accuracy: 71.875000%\n",
      "test_avarage_loss: 0.035076, accuracy: 61.210000%\n",
      "end_time:  2021-11-27 22:20:28 \n",
      "\n",
      "start_time 2021-11-27 22:20:28\n",
      "Train Epoch5 \t Loss: 1.212627, accuracy: 59.375000%\n",
      "test_avarage_loss: 0.032966, accuracy: 63.650000%\n",
      "end_time:  2021-11-27 22:21:02 \n",
      "\n",
      "start_time 2021-11-27 22:21:02\n",
      "Train Epoch6 \t Loss: 1.044114, accuracy: 60.937500%\n",
      "test_avarage_loss: 0.033287, accuracy: 63.620000%\n",
      "end_time:  2021-11-27 22:21:37 \n",
      "\n",
      "start_time 2021-11-27 22:21:37\n",
      "Train Epoch7 \t Loss: 0.954117, accuracy: 67.187500%\n",
      "test_avarage_loss: 0.032278, accuracy: 64.280000%\n",
      "end_time:  2021-11-27 22:22:11 \n",
      "\n",
      "start_time 2021-11-27 22:22:11\n",
      "Train Epoch8 \t Loss: 0.942295, accuracy: 62.500000%\n",
      "test_avarage_loss: 0.031400, accuracy: 65.930000%\n",
      "end_time:  2021-11-27 22:22:46 \n",
      "\n",
      "start_time 2021-11-27 22:22:46\n",
      "Train Epoch9 \t Loss: 0.879152, accuracy: 62.500000%\n",
      "test_avarage_loss: 0.031979, accuracy: 65.020000%\n",
      "end_time:  2021-11-27 22:23:20 \n",
      "\n",
      "start_time 2021-11-27 22:23:20\n",
      "Train Epoch10 \t Loss: 0.922162, accuracy: 59.375000%\n",
      "test_avarage_loss: 0.030523, accuracy: 67.090000%\n",
      "end_time:  2021-11-27 22:23:57 \n",
      "\n",
      "start_time 2021-11-27 22:23:57\n",
      "Train Epoch11 \t Loss: 0.833670, accuracy: 68.750000%\n",
      "test_avarage_loss: 0.030126, accuracy: 67.000000%\n",
      "end_time:  2021-11-27 22:24:32 \n",
      "\n",
      "start_time 2021-11-27 22:24:32\n",
      "Train Epoch12 \t Loss: 0.513380, accuracy: 84.375000%\n",
      "test_avarage_loss: 0.031889, accuracy: 66.270000%\n",
      "end_time:  2021-11-27 22:25:06 \n",
      "\n",
      "start_time 2021-11-27 22:25:06\n",
      "Train Epoch13 \t Loss: 0.901887, accuracy: 67.187500%\n",
      "test_avarage_loss: 0.032048, accuracy: 66.700000%\n",
      "end_time:  2021-11-27 22:25:40 \n",
      "\n",
      "start_time 2021-11-27 22:25:40\n",
      "Train Epoch14 \t Loss: 0.707717, accuracy: 68.750000%\n",
      "test_avarage_loss: 0.031063, accuracy: 67.070000%\n",
      "end_time:  2021-11-27 22:26:15 \n",
      "\n",
      "start_time 2021-11-27 22:26:15\n",
      "Train Epoch15 \t Loss: 0.936380, accuracy: 68.750000%\n",
      "test_avarage_loss: 0.030183, accuracy: 68.300000%\n",
      "end_time:  2021-11-27 22:26:50 \n",
      "\n",
      "start_time 2021-11-27 22:26:50\n",
      "Train Epoch16 \t Loss: 0.544531, accuracy: 82.812500%\n",
      "test_avarage_loss: 0.031889, accuracy: 67.230000%\n",
      "end_time:  2021-11-27 22:27:25 \n",
      "\n",
      "start_time 2021-11-27 22:27:25\n",
      "Train Epoch17 \t Loss: 0.574318, accuracy: 76.562500%\n",
      "test_avarage_loss: 0.031828, accuracy: 67.140000%\n",
      "end_time:  2021-11-27 22:28:00 \n",
      "\n",
      "start_time 2021-11-27 22:28:00\n",
      "Train Epoch18 \t Loss: 0.652767, accuracy: 71.875000%\n",
      "test_avarage_loss: 0.031157, accuracy: 67.190000%\n",
      "end_time:  2021-11-27 22:28:34 \n",
      "\n",
      "start_time 2021-11-27 22:28:34\n",
      "Train Epoch19 \t Loss: 0.723173, accuracy: 78.125000%\n",
      "test_avarage_loss: 0.030779, accuracy: 67.440000%\n",
      "end_time:  2021-11-27 22:29:09 \n",
      "\n",
      "start_time 2021-11-27 22:29:09\n",
      "Train Epoch20 \t Loss: 0.659028, accuracy: 68.750000%\n",
      "test_avarage_loss: 0.030969, accuracy: 67.760000%\n",
      "end_time:  2021-11-27 22:29:44 \n",
      "\n",
      "Finished Training\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAACSCAYAAABsboAjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAmKUlEQVR4nO3deXhU1fnA8e+Zyb4RQjJZSQJJSAJh33dQQcAFsNq61L21ttTduv9arVWrba221GrdcMcVEAQF2UFUQgIECCQBAtkXluzrzPn9kSEGyDIJSWYC7+d55iG5M/feN4fknTPvPfccpbVGCCGE4zLYOwAhhBCtk0QthBAOThK1EEI4OEnUQgjh4CRRCyGEg5NELYQQDk4StRBCODhJ1KJHU0plKqUusXccQnQlSdRCCOHgJFGL845SylUp9ZJSKtf6eEkp5Wp9zl8ptUIpdVIpdVwptVkpZbA+97BSKkcpVaaUOqCUuti+P4kQDZzsHYAQXeBxYBwwDNDAMuAJ4P+AB4BsIMD62nGAVkrFAr8HRmutc5VSkYCxe8MWonnSoxbnoxuAP2utC7XWRcBTwI3W5+qAYCBCa12ntd6sGya8MQOuwECllLPWOlNrfdAu0QtxBknU4nwUAhxp8v0R6zaAvwEZwGql1CGl1CMAWusM4F7gSaBQKbVYKRWCEA5AErU4H+UCEU2+D7duQ2tdprV+QGvdH7gSuP9ULVpr/aHWepJ1Xw08371hC9E8SdTifOCslHI79QA+Ap5QSgUopfyBPwLvAyilLldKRSulFFBCQ8nDopSKVUpdZL3oWA1UARb7/DhCnE4StTgfrKQhsZ56uAGJwG4gBUgC/mJ9bQzwLVAObANe0Vqvp6E+/VegGMgHTMCj3fcjCNEyJQsHCCGEY5MetRBCODhJ1EII4eAkUQshhIOTRC2EEA5OErUQQji4Lpnrw9/fX0dGRnbFoYUQ4ry0Y8eOYq11QHPPdUmijoyMJDExsSsOLYQQ5yWl1JGWnpPShxBCODiHSdRmi+a7jGLSCsrsHYoQQjgUh0nUCrj9nUQW/5hl71CEEMKhOEyiNhgU0SYv0gulRy2EEE05TKIGiDZ5kVFYbu8whBDCoThcos4rqaasus7eoQghhMNwqEQdY/IC4GBRhZ0jEUIIx+FQiTramqjTZeSHEEI0cqhEHe7ngYvRIHVqIYRowqEStZPRQP8AT0nUQgjRhEMlaoAokxfpkqiFEKKRwyXqGJMXWScqqa4z2zsUIYRwCA6YqL3RGg4WSa9aCCHAARP1qZEfUqcWQogGbSZqpVRfpdR6pdQ+pdRepdQ9XRlQpL8HRoMivUAStRBCgG3zUdcDD2itk5RS3sAOpdQarfW+rgjI1clIRB8P6VELIYRVmz1qrXWe1jrJ+nUZkAqEdmVQ0QEyOZMQQpzSrhq1UioSGA780CXRWMUEepF5rJLaektXnkYIIXoEmxO1UsoL+By4V2td2szzdyilEpVSiUVFRecUVIzJG7NFc+SYzPkhhBA2JWqllDMNSfoDrfUXzb1Ga/0/rfUorfWogIBm12e0WeOcH1KnFkIIm0Z9KOBNIFVr/WLXhwRRAV4ohYz8EEIIbOtRTwRuBC5SSu20PuZ0ZVDuLkbCeruTITe9CCFE28PztNZbaFjSsFtFB3jJdKdCCIED3pl4SkygN4eKKzBbtL1DEUIIu3LYRB1t8qK23kLW8Up7hyKEEHbl0IkaZOSHEEL0gEQtdWohxIXNYRO1j5szgT6uMueHEOKC57CJGhruUJRELYS40Dl0oo42eZFRWI5FRn4IIS5gDp+oK2vN5JVW2zsUIYSwG4dO1DGnLijKjS9CiAuYYyfqQG9AluUSQlzYHDpR+3m64OfpIolaCHFBc+hEDQ11arnpRQhxIXP4RB1japicSWsZ+SGEuDA5fKKONnlRWl1PUVmNvUMRQgi7cPhEHWOSC4pCiAub4yfqQJmcSQhxYXP4RG3ydsXb1UkmZxJCXLAcPlErpYgO9LJr6aPObOHGN3/gk+1ZdotBCHHhcvhEDQ0jP+yZqBf/eJTN6cV8tP2o3WIQQly4ekSijjZ5UVxey4mK2m4/d1l1HS99m47RoNiVdZKSyrpuj0EIcWHrEYm6ceSHHVYlf23jIY5V1PLEZfFYNGzJKO72GIQQF7YekagbV3sp6N5EnV9SzRtbDnHF0BBuHBeBj5sTG9MKuzUGIYToEYk61Ncdd2djt4/8+MfqA1gs8NClsTgZDUyK8WdTWrHcJdkOb245zFb5FCLEOekRidpgUESZPLv1gmJqXimfJWVz0/gI+vp5ADB1QAD5pdWktbNnX2e2sOFAIeYLbAGE4vIanvlqHy99m2bvUITo0XpEoobuX5bruVX78XZ14vcXRTdumzIgAIBNaUXtOtZ7245wy9vbeeHr/Z0ao6NbvbcAi4ako3IRVohz0WMSdbTJi7ySasqqu/4PfnN6EZvSirjrohh8PVwatwf3cifG5MWm9PYl6i+Ss3EyKF7bdIhPEi+csdir9uTh7mzEbNFszmhfmwkhftKjEjXA/vz216kPFpVz/8c7uf7179mVdbLV15otmmdX7iestzs3TYg46/mpAwL44fBxqmrNNp07vaCMPTmlPDwrjknR/jy+JIUfDh1r98/Q05yoqOW7g8e4aXwEvh7OrN8viVqIjuoxiXpwaC9cnQz88o0fePiz3aTmlba5z8Gicu77eCczXtzIyj15pBWUM/+VrTy3KpXquuYT7ZLkHFLzSvnDpbG4OhnPen7KgABq6y18f9i2ZPtFcg5Gg2L+iFD+c8MI+vp5cOf7OzhyrMKm/XuqNakFmC2ay4eEMCUmgI1phbJIsRAd1GMSdYivOyvumsTPRoaxbFcOs1/ezC9e28bXe/KoN1tOe21GYTn3Lk5mxosb+XpPPr+e3J8tD1/Eugencs3Ivry28RBz/rWZHUeOn7ZfdZ2Zf6w+wJCwXlwxJKTZOMb088PVyWBTndpi0SxLzmFKjD/+Xq70cnfmrZtHY9Fw+zuJlHZDGcdevt6TT1hvdxJCfZgeF0BxeS17ckvsHZYQPVKPSdTQsIbis/MH8/2jF/PYnDiyT1Rx5/tJTP3bBl7deJCdWSe5Z3EyM/65kW/2FvDryf3Z/PB0Hp0Tj7+XKz5uzjx/9RDeu30MNXUWrn51G39evq+xjPHmlsPklVTz2Jx4DAbVbAxuzkbG9e/DRhsS9feHj5FbUs38EWGN2yL9PXn1lyPJLK7g9x8mn/Umcz4ora5jc3oRsxOCUEoxJSYApZDyhxAd1KMS9Sm+Hi7cMSWKTQ9N57UbRxLu58FfV+1n3n+2snpvAXdMOT1Bn2lyTADf3DeFG8aG89bWw8x6eRNf78njvxsOckm8iXH9+7R6/ikDAjhUVEHW8cpWX7ckKQcvVydmDgw8bfv4qD78ZV4Cm9KK+MtXqe1vAAe3LrWQOrNmVkIwAH28XBka5ssGuVlIiA5xsncA58JoUFw6KIhLBwWxP7+UxMwTzE4Iok8zyflMXq5O/GXeYC4bHMLDn+/mzveTMBoUj8yOa3PfqQMCeBrYlF7EDWPPvuAIUFVrZtWefGYnBOHmfHat+9ox4aQXlvPmlsNEmby4cVzzx+mJVqbkEeTjxvC+vo3bpsUG8PLadI5X1OLn6dLyzkKIs/TIHnVz4oJ8+OW4CJuSdFPjo/rw9b2T+d20KB6fE0+0dV6R1kQFeBLq695qnXpNagHlNfXMHxHa4msemxPP9NgAnvxyL1vSz4+79ypq6tmYVsSshKDTykfTY01o3f4x6EKI8yhRnwsPFycemhXHbZP62fR6pRRTBvjzXcYx6lqoMS9Jyia4lxvj+rVcRjEaFP+6bjhRAZ789oMd7M9veySLo1t/oJCaeguzE4JO2z44tBd9PF1Yf6D7yh/pBWUkHT3RbecToqtIou6gqQMCKKupJ/noybOeKyqrYVN6MXOHhbZ4UfIUbzdn3rplNO7ORm5+60eyT7Re93Z0q/bk4+/lwqhIv9O2GwyKqbEBbEwr6pZb6avrzNzy9nZueetHKmvru/x8QnQlSdQdNCHaH6NBNftRfvmuXMwWzVWtlD2aCuvtwTu3jaGy1sxNb/1ol3m3O0N1nZn1+wu5dFAQxmbeoKbHmjhZWcfONm466gxvbjlMzskqSqvrWZKc0+XnE6IrSaLuIB83Z4b39W32dvIlyTkMCvFhQGDb9e5T4oN9eOOmUWSfqOLWRdt7ZC9wY1oRlbVmZltHe5xpSkwABgUbu7j8UVhazSvrM5gxMJCEUB8Wbc2UGQ9FjyaJ+hxMHRBASk4Jx8prGrdlFJaRklPC/OG29aabGtu/D/+6dhi7s0+y4IOkFuvfjmpVSh69PZwZ29+v2ed7eTgzMqI36w907QXFv68+QK3ZwuNz4rl1Qj/SC8tlwQfRo0miPgdTBgSgz1j15YukHAwKrhzW/J2NbZmVEMzT8xJYf6CIRz5P6TE9wZp6M2tTC5kxMBBnY8u/VtNiTaTklFBYVt0lcezJKeHTHdncOrEfkf6eXD40GH8vFxZtzeyS8wnRHSRRn4OE0F709nBmo7WHaLFolu3MZXJMACZvtw4f94axEdx7SQyfJ2XzwjcHOivcLrU1o5iymnpmD26+7HHKtNiGqWI3dkGvWmvN0yv20dvDpXF6WlcnI9ePjWDdgUIyi8/v+VXE+avNRK2UekspVaiU2tMdAfUkRoNickwAm9KLsVg0Pxw+Ts7JKpsvIrbmnotjuH5sOP/dcJC3thzuhGi71sqUfLzdnJgY5d/q6wYG+2DydmVDFyTqb/bm88Ph49w/YwA+bs6N2385Nhwng+KdbZmdfk4hTnl3WyZ3f5RMTb1tM2u2hy13Ji4CFgLvdvrZzwNTBgTw5a5cUvNLWZKcjaeLkZkDg9resQ1KKZ6em8Dx8lr+vGIfWScqcXUyUl1npqrWTFVdw+PU974ezkT28STS35N+/g3/Bvu4tTk8sDPUmS2s2VfAjPhAXJxaf+9XSjE91sRK62RaTq2USdqjpt7MMytTiQ305trRfU97zuTjxmWDg/k0MZv7ZwzAu0kSF6IzaK15d9sRvN2cmp1181y1mai11puUUpGdfubzxJSYhh7kN3sLWJWSz6yEYNxdOuc/ymhQvHTtMO54bwdvb83ExcmAu7Ox4eFixM3ZiLuzATdnI1nHq9icXkxN/U8XIF2dDET2aUjcswcHMTshuM1E2hHbDh6jpKqOWQm2vUFNjwvg48Qsko6eZEy/5i88ttfbWzPJOl7Fe7ePaTb53zqxH0t35vKZtX4tRGdKOnqCjMJynv/Z4C45fo+e68MRmHzciA/24fVNh6iqM3dK2aMpN2cj7942BrNFNzs2uSmLRZNfWk1mcQWHj1U0/FtcSUpOCV/vzedpr1SuHxvODWPDCfRpu4ZeXF7D5vQiCktrmDkoiH7+ns2+btWefDxcjI1LlbVlYrQ/TgbF+gOFrSbq4xW1fLkzh0sTggju5d7i64rKali4LoNL4k1Mjmk+hqF9fRkR7ss732Vy8/jIbvmkIS4ci3/MwtPFyOUtTI98rjotUSul7gDuAAgPD++sw/YIUwb4k5pXSpCPW5sz73VUW0kaGu7+C/F1J8TXnQnRP9WKLRbNpvQi3vkuk3+vS+eV9RnMHhzMLRMiGBHeG6Uajl1vtrAz6yQbDhSxMa2IlJyf5o9+btV+hvX1Zd6wEC4fGtI4K6HZolm9N5+L4kzNTj7VHG83Z0ZH+rF+fyEPz2p+EqzEzOPc9VEyeSXVPLMylZ+NCOPOqVFENvNm8eKaA1TXmXlsTnyr571lYj/u/iiZDWmFXBQX2OprRc9UWFaNm7PxtGsUXa2suo4Vu/OYNzwET9eu6ft22lG11v8D/gcwatSonjGmrJNMjQngtY2HmDs8xKaE2t0MBsW0WBPTYk1kFlfw3vdH+CQxi+W7ckkI9WHO4GD25JSwOb2Ysup6jAbFiHBfHpw5gKkDTPTxcmHF7lyWJufy5PJ9PP1VKpNj/Jk/PBRvNyeOVdS2eJNLS6bHBfDsyv3klVSd1lu2WDSvbz7EC98cINTXnbduGcX6/UV8nJjFJ4lZXDE0hN9NiyY2qOFmor25JSzensVtE/vRP8Cr1XPOTggi0MeVt7dmSqI+D1XXmZm7cCterk58dffkdpf59uSUcPfiZP529VBGRvS2eb/lu/KoqjPzi9Fd10FVtozTtdaoV2itE2w56KhRo3RiYuI5htZzmC2aNzYf4mcjw5qd/9oRVdQ03Fr97rZM0grKCfRxZdoAE1NjA5gY5U8vj+Z7JAfyy1i6M4cvd+aSc7IKADdnAzuemNGu3kR6QRkz/rmJ564azHVjGn7BT1TU8sCnu1i3v5A5g4P468+GNPaMCkureWPLYd7//giVtWZmDAxkwfRo/roqlQP5ZWx4cHqLMTe1cF06f1+dxpr7phDTjjtHzycnK2v5/tBxm68ptEed2cKCD5L45bgIm0thneW9bZn837K9ADwwYwB3XRxj8751ZgtzF25lX14poyJ68+md4xs/abZl7sIt1NRbWHXPZJv3aY5SaofWelRzz7X5l6WU+giYBvgrpbKBP2mt3+xwNOcho0Hxm6lR9g6jXTxdnfjluAhuGBtOYVkNJm9Xm37JYoO8eXhWHH+YGcv2zON8uSuXfv6e7f7IF23yItTXnfX7C7luTDg7jpzgrg+TKC6v5c9zB3HjuIjT4jH5uPHYnHh+OzWKRd9lsui7TNbs2wrA03MH2ZSkAa4bE86/1mWw6LtMnpnfNRd+HN3La9N5e2smq+6ZTHywT6cee21qIav3FVBQWt2tibqm3swrGw4yKqI3gT5u/Ht9BpcNCW7zU9Ypr28+xL68UmYODGT1vgI2pRcz1Yb49+WWsiu7hD9dMfCcknRb2vxsoLW+TmsdrLV21lqHSZI+vyilCPRxa/cvmcGgGNu/D8/MH8yvJvfv0HmnxQawNaOY/244yC9e24aT0cDnv53ATeMjW4ynt6cL980YwNZHLuKR2XFcNya8sUduiz5erswbFsIXSTmUVNpvzcoVu3P555q0br/ztN5sYfmuXACW7czt9OMv3n4UgF3ZJSR34xSzn+/IIa+kmrsvjuFPVw7E1cnAY0tsu7P3YFE5L32bzuyEIBZeP4JQX3deXH3Apn0/SczCxcnQoSkj2kPuTBR2Mz3WREWtmee/3s+MgYGsuHsSg8N62bSvl6sTd06N4rmrBrd7LPYtE/pRVWfm48SjHQn7nFXW1vPE0j28vDad978/0q3n3pJRTHF5Lb09nPlyZ06nrgyfc7KKjWlF3D6pH16uTiz6LrPTjt2aOrOF/6zPYFhfXybH+GPyduPR2fF8f+g4n+7IbnVfi0XzyOe7cXMy8NTcQbg4Gbjn4hh2ZZfwbWrrk4dV15n5IimbWYOC8PXo2lWLJFELu5kY7c+02ACeunIQr9wwotuu1A8M8WFsPz/e+e6IXRYX/mR7Ficr64gL8ubpFamkZHff6uxLk3Po5e7Mo3PiyS2pZnvm8U479sfbswC4dWIk14wKY2VKXpfN6dLUkqQcck5Wcc/FMY2fxK4d3ZfRkb155qtUiptMmnamD348yvbME/zf5QMbp324akQokX08eHFNWqtvZN/szae0uv6sG6y6giRqYTfuLkYW3TqGmye0XOroKrdOjCTnZBXvf3+Eiprum1K23mzh9c2HGR3Zmw9/PY4+Xi4s+DCJ0uquL8NU1NTzzd4CLhsSzGWDg3F3NrJsV+eUP8wWzaeJWUyJCSCstwc3jY+kzqz58If2fWo5XlHLkuRsm3v69WYLC9dnMDi0V+M8MtBQmnvuqsFU1tbzlxX7mt0352QVf13ZMILp6pFhjdudjAbuuSSG1LxSvt6b3+K5F/+YRbifR5cNyW1KErW4IM0YGESMyYsnl+9jyFOrueLfW3hq+V6+2p1HQWnX9QK/Sskj52QVv5kShZ+nCwuvH07uySoe+nR3l9erV+/Lp6rOzPzhoXi6OjFzUCArU/KorT/3TxUb0wrJK6nmujENvct+/p5Miw3ggx+Otuv4D322m/s+3mXzZGTLduZy9HgldzfpTZ8SbfLmt9OiWbozl41nLPChteaJJSlYNDw7f/BZ+145NJRokxcvrklrdkWizOIKth06xi9G9+2Wm6ckUYsLktGgWLpgIotuHc3vpkXh6Wrkox+PsuDDJMY+u5bJL6zj/k92Ng5B7Axaa17beIhokxcXxZkAGBnhx8Oz4vh6bz5vd/FUrEuScwnr7c7I8IYxwvOGhXKysu6sJNYRH/6Qhb+XKxfH/zQ+/ZYJkRSV1bBqT55Nx1ibWsC3qQVEBXjy6saDfPRj671xs0WzcH0G8cE+XBJvavY1v5sWRX9/T55YmkJV7U+TJX25K5f1B4r4w6Wx9PXzOGs/o0Fx7yUxZBSWN158berjxCwMitN64l1JErW4YHm6OjEt1sQDM2NZfMd4Up68lKULJvLEZfEMCu7FypQ8HvsipdPOtyWjmH15pdwxpf9pvbBfTe7HJfGBPLcqtcuWKSssq2ZLehHzmqzjOSnGHz9PF5buPLelygpKq1l/oJCrR4adNhf5lJgA+vl72vQGVF1n5snle4kxefHV3ZOZOiCAJ5buafVNZMXuXA4XV3DPxdEtls7cnI08e9Vgso5X8dLaNACOldfw5Jd7GdbXl5snRLZ4/DkJwcQFefPy2vTTrmXUmS18tiObi+JMNk3F0BkkUQth5Ww0MKyvL7+a3J9XbxzJgzNj2ZhW1Oy6mB3x6saDBPq4MveMRSWUUvzjmqEE+rix4IMkTlZ2/pqZy3flYdEwb/hP53Y2GrhscDDf7iug7Bxq5J8mZmG26LMuqhkMipvGR7Az6yS72ngDemXDQbKOV/HnuQm4ORtZeP1wYkxeLPggidS80rNeb7Zo/r0ug9hA7zZnqxzXvw8/HxXGG5sPsy+3lKeW76O8pp4Xrh7S6p3EBoPi/hkDOFxcwRdN1t1cv7+QorKaLr0T8axYuu1MQvQwN46PINzPg2dXpp7zyukp2SVszTjGbRP7NTsNZi8PZ/5z/QgKy6p58NNdnV6vXpqcw+DQXkSbTr8bc97wEGrqLazeW9Ch41osmsXbs5gQ1afZeViuHhmGp4uRd1oZqpdZXMGrGw8yd1gI46MaLsx5uznz9q2j8XQ1ctui7WddN1i1J4+MwnLuujjaphrxY3Pi8XV35lfvbOfLXbn8fnqMTWuazhgYyJCwXrz8bXpjrf3j7VmYvF2ZHtt9N/RIohaiBa5ORh6eFcf+/DI+25F1Tsd6bdNBvF2duG5sy72woX19eWxOPN+mFvL65kPNvkZrTVl1HbntqJ23to7niPDe9PVz73D5Y0tGMdknqri2hZuOvN2cuXpkGMt351JUdvYwOa01f/pyL65GA4+fMalWcC933rx5NCVVddy2aHvj6ByLRfPvtRnEmLyYY+McM74eLvzxioHkllQTG+jNb6fZdiexUor7Zgwg52QVnyRmkV/SUOa5ZlRYp82lbguZ5lSIVswZHMTIiN78fXUalw/p2OxoR49VsjIlj19P6d/mWPFbJkTy4+HjPP/1AU5W1lFeU09RWQ2FZTUUWR9VdQ0XxW6b2I8/XjGwzfMvTc7FaFBcMfTsKTiVUswdGsorGzIoLKtu9xJyi7cfpbeHM5cOanmSq5smRPLOtiN89ONR7j5j/o1v9uazMa2IP14+EFMz9d6E0F785/oR3P7Odu7+KJn/3TSKNfvyOVBQxsvXDmvXiIsrh4ZQUWNmfFSfdk3YNG1AACPCfVm4LoOC0mosGn4+quvHTjclPWohWqGU4vHL4ikqq+G1Tc33ctvyxpZDGA2K22xYsEApxfNXDyE6wItXNhxk2c5c0gvLcTEaGB7uyw1jw3l0dhzzhoXw1tbDrEppfUSFxaJZujOHSdH+BHg3P2HY3GEhWDSs2GXb6IxTispqWL23gKtGhLW6qklUgBdTBgTw/vdHqGtyUa6ytp4/L99HXJA3N42PaHH/6XEmnrpyEGv3F/LU8r28vDaD/v6e7Z77WSnF9WPDW5xXvbX9HpgZS35pNf9Zn8GEqD5E9GnfMc6V9KiFaMOI8N5cNiSY/206yPVjwgnqZXuv81h5DZ8kZjF/eKjNIwR83JxZec9k6syWFuf4rq23cPhYJQ99tptBIb0I73P2EDOAHUdPkH2iigdnxrZ4vphAbwYG+7BsVy63TbJ99ZvPk7Kpt+jGsdOtuWVCBLctSmTVnnyutPbs/70ug9ySal6+bnibZYQbx0eSeaySN63rh77486HdOqXwhKg+jO3nxw+Hj/OLbrgT8UzSoxbCBo/MisNigX+sbt+q8O9uO0J1nYU7prRv4iqjQbW6EIOLk4GF1w1HKVjwYVKLC6ouSc7Bw8XIzFZKE9BwUXFX1kkO27hSu9aaj7dnMTqy91kXKJszbYCJiD4ejRcVMwrLG6YGHhHG6EjblmN7bE48c4eFMDi0V2Oy7y5KKZ68chBXjwzj0kGdPz1sWyRRC2GDvn4e3DIxks+Sstmba9vcHJW19by7LZNL4gNtSmYdienv1wwlJaeE51buP+v5mnozX+3O49JBQXi4tP7h+YqhISgFy2y8qPj9oeMcLq7gWhuHqDUM1Ytkx5ETpGSX8Mdle3B3NvLonOZX+GmO0aB4+drhLFswsVsv5J0SH+zD368ZavNKRp1JErUQNlowLZpe7s48uzLVpuFznyZmc6Kyjjuntn8aWFvNHBTE7ZP6sei7zLPq1RsOFFFSVcc8G6bgDO7lzth+fizbmWvTz7Z4+1G83ZyYM9j2lX2uGRWGh4uRuz5K4ruDx/jDpbEdWmjjQlzvUhK1EDbq5eHMPRfHsDXjGBsOtH4TTMPkS4cYGdGbUTZ+tO+oh2fFMbSvLw99tpujxyobty9NzsHfy5WJUbZNGjRvWCiHiyvY3cZsficqalmVks9Vw0Nxd7G9d+nj5szPRoSReayShFAfrh/b8gVEcTpJ1EK0ww1jI+jn78kzK1ObnSK1pLKOL3flsuDDJLJPVPGbdtamO6K5enVJVR1rUwu5cmiIzWWC2YODcTEa2lxQ4IvkHGrNlhbHTrfm9kn9GBDoxbPzBzvk+qKOSkZ9CNEOLk4GHpkdx2/e28Hi7VncMDac/fllrNtfyIYDhew4cgKLht4eztwyIZJL4rtnEd2+fh787Zqh/Oa9HTy3cj9xQd7Umi3tWnmkl7sz0+MCWL47l8cviz8tkVbXmVm/v5AVu/P4NrWAoX19O7SMV6S/J6vvm9ru/S50kqiFaKeZAwMZE+nHC1/vZ+G6DPKttzcnhPqwYHo00+NMDA3z7fYe46WDgrhtYj/e2noYk7crUQGeJIS2L5nOHRbKN3sL+O5gMWP6+bE5rZgVu3NZs6+AilozfTxd+Pmovvy6A8uviY6TRC1EOyml+OMVA/ndB0kMCvFheqyJabEBzd5Z190emR3HjiPH2ZVdwk3jI9q9IMNFcSa8XZ34v6V7OFZRS1l1Pb4ezlwxNITLh4Qwrr+fXUZcXOgkUQvRAQmhvdj00HR7h3EWFycDC68fwcJ1GR2qIbs5G7l6VBifJWYzc1AQlw8NZlK0/2nTl4rup7piVYlRo0bpxMTETj+uEKJ7aK27fXm0C51SaofWelRzz8nbpBDiLJKkHYskaiGEcHCSqIUQwsF1SY1aKVUEHOng7v5AcSeG05kkto6R2DpGYuuYnhpbhNa62WVjuiRRnwulVGJLBXV7k9g6RmLrGImtY87H2KT0IYQQDk4StRBCODhHTNT/s3cArZDYOkZi6xiJrWPOu9gcrkYthBDidI7YoxZCCNGEwyRqpdQspdQBpVSGUuoRe8fTlFIqUymVopTaqZSy+73xSqm3lFKFSqk9Tbb5KaXWKKXSrf/2dqDYnlRK5Vjbb6dSao4d4uqrlFqvlNqnlNqrlLrHut3u7dZKbI7Qbm5KqR+VUrussT1l3d5PKfWD9e/1Y6WUiwPFtkgpdbhJuw3r7tiaxGhUSiUrpVZYv+9Yu2mt7f4AjMBBoD/gAuwCBto7ribxZQL+9o6jSTxTgBHAnibbXgAesX79CPC8A8X2JPCgndssGBhh/dobSAMGOkK7tRKbI7SbArysXzsDPwDjgE+Aa63bXwV+60CxLQKutme7NYnxfuBDYIX1+w61m6P0qMcAGVrrQ1rrWmAxMNfOMTksrfUm4PgZm+cC71i/fgeY150xndJCbHantc7TWidZvy4DUoFQHKDdWonN7nSDcuu3ztaHBi4CPrNut1e7tRSbQ1BKhQGXAW9Yv1d0sN0cJVGHAllNvs/GQX5RrTSwWim1Qyl1h72DaUGg1vrU6qb5QPcsLWK73yuldltLI3Ypy5yilIoEhtPQA3OodjsjNnCAdrN+fN8JFAJraPj0e1JrXW99id3+Xs+MTWt9qt2esbbbP5VS7V9Bt3O8BDwEnFqzrQ8dbDdHSdSObpLWegQwG1iglJpi74Baoxs+VzlMzwL4LxAFDAPygH/YKxCllBfwOXCv1rq06XP2brdmYnOIdtNam7XWw4AwGj79xtkjjuacGZtSKgF4lIYYRwN+wMPdHZdS6nKgUGu9ozOO5yiJOgfo2+T7MOs2h6C1zrH+WwgsoeGX1dEUKKWCAaz/Fto5nkZa6wLrH5QFeB07tZ9SypmGRPiB1voL62aHaLfmYnOUdjtFa30SWA+MB3yVUqcWHrH732uT2GZZS0laa10DvI192m0icKVSKpOGUu5FwMt0sN0cJVFvB2KsV0RdgGuBL+0cEwBKKU+llPepr4GZwJ7W97KLL4GbrV/fDCyzYyynOZUIreZjh/az1gffBFK11i82ecru7dZSbA7SbgFKKV/r1+7ADBpq6OuBq60vs1e7NRfb/iZvvIqGGnC3t5vW+lGtdZjWOpKGfLZOa30DHW03e18VbXJ1dA4NV7sPAo/bO54mcfWnYRTKLmCvI8QGfETDR+E6Gupct9NQ/1oLpAPfAn4OFNt7QAqwm4bEGGyHuCbRUNbYDey0PuY4Qru1EpsjtNsQINkawx7gj9bt/YEfgQzgU8DVgWJbZ223PcD7WEeG2OsBTOOnUR8daje5M1EIIRyco5Q+hBBCtEAStRBCODhJ1EII4eAkUQshhIOTRC2EEA5OErUQQjg4SdRCCOHgJFELIYSD+3+xwMcuU0B8RQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAACSCAYAAABVCTF4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAkg0lEQVR4nO3dd3iUVfrw8e+dHkoSQkJLIZRQIjX0jtgoa9dVFBsKrq5tLavuqsu6689XXXV1da0IggU7oqKi0nsvCTUJhIQAaaT3zHn/mIEdQhKGkGQyk/tzXXMx8zxnZm6PmXvOnHOec8QYg1JKKdfn4ewAlFJK1Q9N6Eop5SY0oSullJvQhK6UUm5CE7pSSrkJTehKKeUmNKErpZSb0ISuXI6ILBeREyLi6+xYlGpKNKErlyIiUcAYwABXNOL7ejXWeylVV5rQlau5FVgPzAVuO3lQRCJE5GsRyRCRLBF5w+7cDBHZIyL5IrJbRGJtx42IdLcrN1dE/mm7P15EUkXkcRE5BswRkTYi8r3tPU7Y7ofbPT9YROaISJrt/ELb8TgRudyunLeIZIrIwIaqJNU8aUJXruZW4GPb7TIRaS8insD3QDIQBYQBCwBE5Hpglu15AVhb9VkOvlcHIBjoDMzE+nmZY3scCRQDb9iVnw+0AC4A2gGv2o7PA6bZlZsMHDXGbHMwDqUcIrqWi3IVIjIaWAZ0NMZkishe4B2sLfZFtuMVVZ7zM7DYGPNaNa9ngGhjTILt8Vwg1RjzlIiMB5YAAcaYkhriGQAsM8a0EZGOwBGgrTHmRJVynYB9QJgxJk9EvgQ2GmNerGNVKFUtbaErV3IbsMQYk2l7/IntWASQXDWZ20QAiXV8vwz7ZC4iLUTkHRFJFpE8YCUQZPuFEAFkV03mAMaYNGANcK2IBAGTsP7CUKpe6UCPcgki4g/8HvC09WkD+AJBwHEgUkS8qknqKUC3Gl62CGsXyUkdgFS7x1V/vj4C9ASGGWOO2Vro2wCxvU+wiAQZY3Kqea8PgbuwfubWGWOO1BCTUnWmLXTlKq4CKoEYYIDt1htYZTt3FPh/ItJSRPxEZJTtee8Dj4rIILHqLiKdbee2AzeJiKeITATGnSWG1lj7zXNEJBj428kTxpijwI/Af22Dp94iMtbuuQuBWOBBrH3qStU7TejKVdwGzDHGHDbGHDt5wzooORW4HOgOHMbayr4BwBjzBfAc1u6ZfKyJNdj2mg/anpcD3Gw7V5t/A/5AJtZ++5+qnL8FKAf2AunAQydPGGOKga+ALsDXjv9nK+U4HRRVqpGIyDNAD2PMtLMWVqoOtA9dqUZg66K5E2srXqkGoV0uSjUwEZmBddD0R2PMSmfHo9yXdrkopZSb0Ba6Ukq5CU3oSinlJpw2KBoSEmKioqKc9fZKKeWStmzZkmmMCa3unNMSelRUFJs3b3bW2yullEsSkeSazmmXi1JKuQlN6Eo1oGO5JaTnV7tYo1L1ThO6Ug3EYjHc9P56bp29EZ0erBqDJnSlGsjy/ekkZRSy91g+y/dlODucRmWMISG9wNlhNDsOJXQRmSgi+0QkQUSeqOZ8pIgsE5FtIrJTRCbXf6hKuZYPVh+ifYAvnQL9eGdlXZdkd01vLkvg4ldW8OOuo84OpVk5a0K3Ld7/JtZF+WOAqSISU6XYU8DnxpiBwI3Af+s7UKVcyb5j+axOyOTWEVFMH92F9UnZ7EjJcXZYjSL1RBFvLEsA4Nnvd1NYWt2+I6ohONJCHwokGGOSjDFlWPdqvLJKGYN1v0aAQCCt/kJUyvXMXXsQXy8PbhoayY1DI2nt58W7K5OcHVajeO6HPQC8PnUgR3NLeH3pASdH1Hw4ktDDsC4sdFKq7Zi9WcA0EUkFFgP310t0Srmg7MIyvt56hGtiw2jT0odWvl5MG96ZH+OOkpxV6OzwGtSqAxn8GHeM+y7szhX9O3H9oHBmrzpIQnq+s0NrFuprUHQqMNcYE451R/P5InLGa4vITBHZLCKbMzKa1yCRaj4+3XiY0goLd4zqcurYHSOj8PLw4P1VB50YWcMqq7Awa1E8ndu24K4xXQF4YlIvWvp68fTCeJ3p0wgcSehHsG6Ae1K47Zi9O4HPAYwx6wA/IKTqCxlj3jXGDDbGDA4NrfbKVaVcWlmFhXnrDjEmOoQe7VufOt4uwI+rB4bx+eYUsgpKnRhhw5m79iCJGYX87fIY/Lw9AWjbypfHLuvJuqQsFu1wjZ7YvJJy4tNynR1GnTiS0DcB0SLSRUR8sA56LqpS5jBwEYCI9Maa0LUJrpqdH+OOcjyvlOl2rfOTZoztQmmFhXnrarxy22UdzyvhtV8PcFGvdkzo1f60c1OHRtIvPJDnfthDfkm5kyJ0zK7UXCa/toopr6/mH9/vpqzC4uyQzslZE7ptF/X7gJ+BPVhns8SLyLMicoWt2CPADBHZAXwK3G7095VqZowxzF59kK4hLRnX48xfoN3btebi3u2Zt+4QxWWVToiw4Ty/eA/llYZnLq86AQ48PYR/XNmHjIJS/v2rYwOkx3JLSMkuqu8wa2SM4aP1yVz71losFmPt+199kOvfWdeocZwvh/rQjTGLjTE9jDHdjDHP2Y49Y4xZZLu/2xgzyhjT3xgzwBizpCGDVqop2nr4BDtTc7ljVBQeHlJtmbvHdeVEUTlfbEmp9rwr2pCUxcLtadw9riud27astkz/iCCmDo1k7tpD7D2WV+NrlVVYeP23A4x9aRnjXlrGX7/Z1eBdVEVlFfzps+08tTCOEd3a8sMDY3jp+v68dXMsSekFTHl9FT/HH2vQGOqLXimqVD35YM0hAvy8uCY2vMYygzu3ITYyiPdWJVFR6Vo/56tTUWnhb4viCQvy597x3Wst++fLehLo783TC+OqHSDddCibya+v4pVf9nNpTHtuHRHFgk0pjH9pOe+sSKS0ov5/1SSk53PlG2v4dkcaj1zSgzm3D6FNSx8AJvXtyA8PjCEqpCV3z9/Cs981/S4YTehK1YMjOcX8FHeMqUMjaelb86rUIsLMsd1IyS7mJxdp9dXm4w2H2Xssn6em9Mbfx7PWskEtfHhiYi82HTrBV1v/N68it6icJ7/exfVvr6O4rJI5dwzhjZtimXXFBfz80FiGdAnm+R/3cskrK/lx19F6my2zaEcaV7yxhuzCMuZPH8b9F0Wf8csqsm0LvvjDCG4fGcUHaw5y/dtrz7sLZvWBzAb7YtCErlQ9mLfuEAC3jow6a9lLYtrTNaQl76xIcumpfJkFpby8ZB+ju4cwsU8Hh55z3aBwYiODeH7xHnKLyvluRxoXvbKCzzYdZsaYLvzy8Fgu7NnuVPnu7Vrxwe1DmH/nUPy9Pbnn463c8O56dqXWfRZK6okinl4YxwOfbiOmYwA/PDCG0dFnTMo7xdfLk1lXXMDb02JJyixkch27YIwxvLsykWmzNzB7dcNMX3XaJtGDBw82usGFcgdFZRUM/7/fGBMdyps3xzr0nE82HOYv3+zikxnDGNmt5mTSlD3+5U6+2prKTw+NoXu71md/gk18Wi6X/2c17Vr7cSyvhL5hgTx/TV/6hAXW+ryKSgsLNqXw6i/7yS4qY2BEEP3Cg+gTFki/8EC6hbbCs5qxi4z8UtYmZrIuMYu1iVkctrWwZ4zpwp8n9sLb0/F2bUp2EX/8ZCs7U3O5Z3w3Hr20Z7XvWVWlxfCP73czd+0hpvTryMvX9z81tfNcicgWY8zgas9pQm88WQWl5JVU0CWk+oEj5Zrmr0/m6YVxfHXPCAZ1DnboOSXllYx+YSkXdArkw+lDGzjC6sUdsbZyz5ZIq8otKuf//bSXTzceZubYrvxlcu9zfu//W7yHj9Yn88ilPbltRGe8ziGp5pWU8/6qg6xLzCQ+LY8i24whf29PLugUQJ+wQHq0b83+4/msTcxk/3Hrqo+t/bwY3rUtI7u1ZUx0yDl9Cdkrrajk79/t5pMNhxndPYTXpw4k2NbvXp2S8koeXLCNn+OPM2NMF56c1LvGQXNHaEJvIm6ZvYHtKTmseOzCWv8AlOuwWAwXv7qC1r5eLPzjKEQc/6C+sfQA/1qynx8fHEPvjgFnf4KdikoLry9NYOG2I0zq04Hpo7vQPsDPoefGp+Xy8pL9LN2bDsCFPUN55NKeZ03sxhgWbj/CP7/fQ05xOdNHRfHIpT3r1NI0xlBWacHXq26t1JMqLYakjAJ2Hcm13lJziU/Lo7i8Ej9vD4ZEBTOyWwijurflgk6BDrWmHfX5phSe+jaO0Fa+vD1tEH3Dz6y/7MIy7vpwE9tScnh6SgzTR595fcK50oTeBCRmFHDRyysAmD6qS7XzdZVrKS6r5MutqTy9MI5/3zCAqwZWXeKodjlFZYx5YRm+3h787fIL+F2/jg59IaRkF/HQZ9vZknyC/uGB7DqSi5eHB9cOCmPm2G41/gJMSC/g1V/288OuowT4eXH3uG6IwDsrksgtLmdy3w48fEmPaluuiRkFPPVNHOuSshgYGcRzV/UlptO5fQk1lkqLIfVEER0D/fHxathhwp2pOdzz0VYyCkr555V9+P2Q/11UfziriNvnbCQ1p5jXbhjApL4d6+U9NaE3AbMWxfPJhsNc2CuUpXvT+fXhcTXO2VWNJyO/lIc/305wSx8u6BRATMdAYjoFVPsLymIxxKXlsupAJqsPZLIl+QRllRa6hrbkpwfH1il57E7L44mvd7IzNZcJvdrxj6v6EBbkX2P573em8eTXuzAGnru6D1cOCONwVhHvrkrki82plFVamNSnA38Y141+4UGA9Qvg378e4Jttqfh5ezJ9VBdmjO1KoL83ALnF5cxelcTs1QcpLq/k6oHhPHRxNBHBLSgpr+S/yxN5e3kift4ePD6pF1OHRJ5Xl4G7yS4s44FPt7E6IZOpQyOZdUUM+47lM33uJsorDbNvG8zgKMe64hyhCd3JCkutg2YX9W7Hk5N7M/6l5VzUux1v3OTYAJqq3d5jefh6edZpbOKxL3bwzbYjtGvtS1ru//b+7BjoR0zHAGI6BRDa2pcNSdmsScwkp8h66XqvDq0ZEx3C6OhQhkYFn3XKXm0qKi3MXXuIl5fsRwQevbQnt42MOq17oKisgr8v2s1nm1MYEBHE6zcOJLJti9NeJyO/lLlrDzJvXTL5JRWM6t6WyOCWfLklBRHhluGduWd8N0Ja+VYbR1ZBKW+vSOTDdckYY7h6YBgbDmaTnFXEVQM68dcpMYS2rv65zV2lxfCvJft4a3kiMR0DOJRVSJsWPnw4fSjd27Wq1/fShO5kH61P5qmFcXx970hiI9vwypJ91v7PP45iQESQs8NzurScYn7bm84NgyPOuZW7JiGTO+Zuok0Lb5b8adypVqcjdqTkcOWba7h7bFeenNyb7MIy9hzNIz4tl91pecSn5ZGYUYDFQLvWvoyODmFMdAijuofQrrVj/dXnIiW7iKcWxrFifwb9wwN5/pp+xHQKIO5ILg8s2MbBzELuHd+Nhy7uUevMjPyScj7deJj3Vx0ku7CM3w+J4P4J3ekYWHPL396x3BL+s/QAn21KISK4Bf+8qg+jurvmTJzG9lPcMR79Yged27Zgzh1DGuTvRBO6ExljuOzfK/Hx8uC7+0YjIhSUVjD+pWV0DW3FZzOHn9NAmjNYLIbtqTmcKCwjwN+bAD9vAvy9CPT3xt/b87ziP3A8n1tmb+RYXgnjeoTy1rRYWvjUfGGOvQ1JWdw2ZyMdA/1JzirkhiERPH9NP4eea4zh2rfWcji7mGWPjqO1X/VfBCXllWTklxLexr9R/j8ZY1i0I41nv9tNTnE5k/p0YEn8cdq09ObVGwac0xTHsgoLxeWV5/QlZ6+gtAJfL49zmtanrLOAWvh6Nli91ZbQHfvkqDrbcDCb/ccLePG6fqcSQitfLx68uAdPL4zj1z3pXBLT/iyv0vgsFsO2lBP8sPMYi3cd5VheSbXlvDyEAH9vgvy9uX5wBDPHdnV4JsG2wye4Y+4mvD09ePCiaP6z9ADT3t/AB7cPIahF7bOAth4+wfS5mwgL8uezu0fw3sok3lmZxOX9OjHSgdbkt9vT2Ho4hxev61djMgfw8/YkIrhFjefrm4hw5YAwxvUI5bkf9vDFllQuiWnPi9f2O3VJuqN8vDzOa1CwVS1XvKqaBbao2xdofdAWegO79+MtrE3MYv2TF502vau80sJlr65EBH5+aOw5zcNtKPZJ/Me4oxzNLcHH04OxPUL5Xb+OdAlpSV5JOXnFFeSVlJNbXE5ecTl5JeUkZRSyNjGLoVHBvPz7/mdNgqsOZHD3/C2EtPLlozuHEdm2BT/FHeWBT7cTFdKCedOH0SGw+p+ru1Jzuen99bRt6cNnd4+gfYAfxWWVTHptJRZjrc/a+rQLSyuY8PJy2gf4sfDeUU16gC8jv5SQVj5N/lecajzaQneSY7kl/Bx/nLtGdzljrq63p3XGwN3zt/D55lRuGhbppCitAzpz1hxk9uqDpyXxP0/syUW92xNQSwv2JGMM32w7wjPfxjP5tVU8e9UFXDUgrNpE9P3ONP702Xa6t2vNh9P/1884sU9H5k73Zua8LVz71lo+umvYGQOde47mccsHGwjw8+bjGcNPzb329/Hk+Wv6MfW99bz66/5aL3Z5a3kix/NK+e/Ng5p0Mgd0EFKdE+c3C93YJxuSsRjDtOGdqz1/aUx7Bnduwyu/7Hfazugp2UVMfW89//xhD11DW/LqDf3Z/PTFvH/bYK4eGO5QMgdrV8E1seH8+OAYenVszZ8+28H9n24jt+j0DQ3mr0/m/k+3MSAiiAUzh58xaDSyWwifzhhOcXkl17219tTVjGBdGW/a+xvw8/Lk0xnDz5jeN6JbW6YOjeT9VUnsSMmpNk7rFL8krhrQiUGd2zj036aUq9CE3kDKKix8sjGFCT3b1dj9ICL8ZUpvMgtKeW/V+e0Ib4z1YopKi2NdaMYYPt+cwqTXVrE7LY+XruvHR3cOO6ckXp2I4BYsmDmCxy7ryU9xx5j42krWJmZijOE/vx3g6YVxTOjZjnnTh9U4WNc3PJAv/zACP29Pbnx3PesSsziUWchN721ARPhkxrAzpuyd9OTkXoS29uXxr3ZWu6Ld/y3eg6cIT0w698vVlWrqtMulgfwYd5TMglJuGVF96/yk2Mg2TO7bgXdXJnHTsMg6TXNKyS7imW/jWLYvg/YBvlw1MIxrY8NP29PSXmZBKU9+vYtfdh9neNdg/nV9f8Lb1N/An6eH8McLuzMmOoSHFmzn5vc3MKRzMBsPZXNNbBgvXNvvrDMAuoa24st7RnDr7I3cNmcjQf7eVFgMC2YOp2tozfN6A/y8ee6qvtw1bzNvr0jkgYuiT51bm5DJT/HHePTSHjX2zyvlyhxqoYvIRBHZJyIJIvJEDWV+LyK7RSReRD6p3zBdz7x1yUS1bcHY6LNvhv3YZb0oq7A4vD3XSeWVFt5ekcglr65gw8Fs7ruwO33Dgpi96iCXvrqS3/1nFR+sPkim3Y4vS+KPcdmrK1mxP4OnpvTmk7uG12syt9cvPIjvHxjNzcMi2XgomztHd+Ff1/V3eDpXx0B/Pr97BBd0CqC0wsL8O4fW+CVl7+KY9lzevxNvLE3gwPF8wHrxzt+/2014G/9TO9Ir5W7OOstFRDyB/cAlQCrWTaOnGmN225WJBj4HJhhjTohIO2NMem2v686zXOKO5PK7/6zmqSm9HU4esxbFM399Mj87uBTpluQT/PWbXew9ls+lMe2ZdcUFdLL1KWcVlPLdjjS+3naEnam5eHoI43uE0tLXi0U70rigUwCv3jDAoeRYX04Ulp3ztLuTKi2GorKKWqcXVpVZUMrFr6yga0hLvvjDSD7ekMwz38bz9rRYJvapnzU1lHKG853lMhRIMMYk2V5sAXAlsNuuzAzgTWPMCYCzJXN3N39dMv7enlw/KOLshW3un9Cdr7akMuX11fQLDyS2cxtiI603+5kOucXlvPjTXj7ZeJgOAX68c8sgLrvg9M0F2rby5fZRXbh9VBcOHM/n621H+GbrEdLzS7h/QnfunxDd4IsWVVXXZA7WLpxzSeYAIa18+dvlMfzpsx28/tsBPlx3iJHd2p5RV0q5E0cSehhgv6NtKjCsSpkeACKyBvAEZhljfqqXCF1MTlEZ3+44wtUDw8/pAoO2rXz5eMYwvt2expbkE3yw+iDvVFoHSiOC/RkU2Yauoa2Yty6Z7MJSpo/qwp8u6XHWiz+i27fm8Ym9ePTSnufcynV1Vw0I49vtabz22wE8BJ65PEbncyu3Vl+Dol5ANDAeCAdWikhfY0yOfSERmQnMBIiMdN6864b0xeZUSsot3HqWwdDq9AsPOrVCXkl5JfFpuWxJPsHW5BzWJlp3Vu8bFsjcO4ac86YEdWnlujoR4bmr+zLl9VVcGxtOrw5Nc7lXpeqLIwn9CGDfdxBuO2YvFdhgjCkHDorIfqwJfpN9IWPMu8C7YO1Dr2vQ5yM5q5CWvl41rjhXm0qLYXVCJgDBLXxo09Kb4JY+p9YesVgM89cnMzQq+Jw3LKjKz9uTQZ2DT+2AY4whq7CM4BY+Tf5imKYkLMiftU9MwL+O230p5UocSeibgGgR6YI1kd8I3FSlzEJgKjBHREKwdsGc38TqBmCM4cZ312MMfH3vyFODiI4+95lv4/h4w+Ezzvl5exDcwocWvl4czi7isct61mfYgLW1WZcvIYXDi30p5erO+pdujKkQkfuAn7H2j39gjIkXkWeBzcaYRbZzl4rIbqASeMwYk9WQgdfF/uMFHLWteX3bBxv58g8jHe7nfmNpAh9vOMxdo7swqW8HsgvLOVFYRnZRmfXfwjJOFJUR0zFAB96UUk7hUNPFGLMYWFzl2DN29w3wsO3WZJ3sLvnX9f35y9e7uGveJubfOeyseyJ+tukwL/+yn2tiw/jrlN46sKaUapKa1aX/qw9k0CWkJdcNCueVG/qzOfkEDy7YVuvl8kv3Hucv38QxtkcoL1zbT5O5UqrJajYJvazCwoaD2Yy2rZX9u36deHpKDD/HH2fWoniqu8Bq2+ET3PvxVmI6BvDWzbG60L9SqklrNqNF21NyKCqrPG0rremju3A8r4R3VibRPsCX+yb8b92PxIwCps/dRPsAPz64fQgtdbF/pVQT12yy1OoDGXiIdYlVe49P7EV6fin/WrKfdgF+/H5wBOl5Jdz2wUY8RJg3faiuSa2UcgnNJ6EnZNIvPOiMJVs9PIQXru13agVCP29P3l6eSHZhGQtmDqdz23PfSV4ppZyhWXQK55WUsyM191T/eVU+Xh68NW0QvTu25oFPt7H/eD7/vTn21FWbSinlCppFQl+fmEWlxZzWf15VK18v5tw+lDHRIbxywwDG92zXiBEqpdT5axZdLmsSMvH39iS2c1Ct5UJb+zL/zqrrjimllGtoFi301QmZDO0SjK+XruehlHJfbp/Qj+YWk5hRWGP/uVJKuQu3T+irD1gv9x8drQldKeXe3D6hr0nIJKSVDz0bcbs1pZRyBrdO6MYYVidkMbJbiK4hrpRye26d0PcdzyezoFS7W5RSzYJbJ/ST/ee1zT9XSil34dYJfU1CJl1DWhJ2DjsTKaWUq3IooYvIRBHZJyIJIvJELeWuFREjIoPrL8S6ObVcrna3KKWaibMmdBHxBN4EJgExwFQRiammXGvgQWBDfQdZF9sOnzhjuVyllHJnjrTQhwIJxpgkY0wZsAC4sppy/wBeAErqMb46W5OQiYfA8K5tz15YKaXcgCMJPQxIsXucajt2iojEAhHGmB/qMbbzsiohk/4RZy6Xq5RS7uq8B0VFxAN4BXjEgbIzRWSziGzOyMg437euUV5JOTtScvRyf6VUs+JIQj8CRNg9DrcdO6k10AdYLiKHgOHAouoGRo0x7xpjBhtjBoeGhtY96rNYn5iFxeh0RaVU8+JIQt8ERItIFxHxAW4EFp08aYzJNcaEGGOijDFRwHrgCmPM5gaJ2AGrTy6XG9nGWSEopVSjO2tCN8ZUAPcBPwN7gM+NMfEi8qyIXNHQAdbF6oRMhnUNxsfLrafZK6XUaRza4MIYsxhYXOXYMzWUHX/+YdVdWk4xSRmF3DQ00plhKKVUo3O7JuzqBF0uVynVPLlfQj+QSUgrX10uVynV7LhVQq+otLBifwbjeoQiosvlKqWaF7dK6NtTcsgtLufCXg03JVIppZoqt0roS/em4+khjInWhK6Uan7cLqEP7txGL/dXSjVLbpPQ03KK2Xssnwm92jk7FKWUcgq3SejL9qUDaEJXSjVb7pPQ96YT3saf7u1aOTsUpZRyCrdI6CXllaxJyGJCr3Y6XVEp1Wy5RUJfn5RFcXklF2p3i1KqGXOLhL5sbzp+3h6M0N2JlFLNmMsndGMMS/elM6pbCH7ens4ORymlnMblE3piRgEp2cXa3aKUavZcPqEv22vdyk4TulKquXP5hL50bzq9OrQmLMjf2aEopZRTuXRCzyspZ9OhbG2dK6UUDiZ0EZkoIvtEJEFEnqjm/MMisltEdorIbyLSuf5DPdPqA5lUWAwX9tSErpRSZ03oIuIJvAlMAmKAqSISU6XYNmCwMaYf8CXwYn0HWp2le9MJ9PcmNjKoMd5OKaWaNEda6EOBBGNMkjGmDFgAXGlfwBizzBhTZHu4Hgiv3zDPZLEYlu9LZ2yPULw8XbrnSCml6oUjmTAMSLF7nGo7VpM7gR/PJyhH7DqSS2ZBGRN0MwullALAqz5fTESmAYOBcTWcnwnMBIiMjDyv91q6Nx0RGNdD+8+VUgoca6EfASLsHofbjp1GRC4G/gpcYYwpre6FjDHvGmMGG2MGh4aeX8t62b50BkYEEdzS57xeRyml3IUjCX0TEC0iXUTEB7gRWGRfQEQGAu9gTebp9R/m6dLzS9iZmqtrnyullJ2zJnRjTAVwH/AzsAf43BgTLyLPisgVtmIvAa2AL0Rku4gsquHl6sXyfXp1qFJKVeVQH7oxZjGwuMqxZ+zuX1zPcdVq2d50OgT4EdMxoDHfVimlmjSXm+9XXmlh1YFMLuwVqptZKKWUHZdL6JsOZVNQWqFXhyqlVBUul9C3HDqBj6cHo7qHODsUpZRqUup1HnpjuP+iaK4fHEFLX5cLXSmlGpTLtdABOgT6OTsEpZRqclwyoSullDqTJnSllHITYoxxzhuLZADJdXx6CJBZj+HUJ42tbjS2utHY6saVY+tsjKl27RSnJfTzISKbjTGDnR1HdTS2utHY6kZjqxt3jU27XJRSyk1oQldKKTfhqgn9XWcHUAuNrW40trrR2OrGLWNzyT50pZRSZ3LVFrpSSqkqXC6hi8hEEdknIgki8oSz47EnIodEZJdtTfjNTo7lAxFJF5E4u2PBIvKLiByw/dumCcU2S0SO2Opuu4hMdlJsESKyTER2i0i8iDxoO+70uqslNqfXnYj4ichGEdlhi+3vtuNdRGSD7fP6mW2TnKYS21wROWhXbwMaOza7GD1FZJuIfG97XLd6M8a4zA3wBBKBroAPsAOIcXZcdvEdAkKcHYctlrFALBBnd+xF4Anb/SeAF5pQbLOAR5tAvXUEYm33WwP7gZimUHe1xOb0ugMEaGW77w1sAIYDnwM32o6/DdzThGKbC1zn7L85W1wPA58A39se16neXK2FPhRIMMYkGWPKgAXAlU6OqUkyxqwEsqscvhL40Hb/Q+CqxozppBpiaxKMMUeNMVtt9/Ox7tIVRhOou1piczpjVWB76G27GWAC8KXtuLPqrabYmgQRCQemAO/bHgt1rDdXS+hhQIrd41SayB+0jQGWiMgWEZnp7GCq0d4Yc9R2/xjQ3pnBVOM+Edlp65JxSneQPRGJAgZibdE1qbqrEhs0gbqzdRtsB9KBX7D+ms4x1m0swYmf16qxGWNO1ttztnp7VUR8nREb8G/gz4DF9rgtdaw3V0voTd1oY0wsMAn4o4iMdXZANTHW33JNppUCvAV0AwYAR4GXnRmMiLQCvgIeMsbk2Z9zdt1VE1uTqDtjTKUxZgAQjvXXdC9nxFGdqrGJSB/gSawxDgGCgccbOy4R+R2QbozZUh+v52oJ/QgQYfc43HasSTDGHLH9mw58g/WPuik5LiIdAWz/pjs5nlOMMcdtHzoL8B5OrDsR8caaMD82xnxtO9wk6q662JpS3dniyQGWASOAIBE5uXmB0z+vdrFNtHVhGWNMKTAH59TbKOAKETmEtQt5AvAadaw3V0vom4Bo2wiwD3AjsMjJMQEgIi1FpPXJ+8ClQFztz2p0i4DbbPdvA751YiynOZksba7GSXVn67+cDewxxrxid8rpdVdTbE2h7kQkVESCbPf9gUuw9vEvA66zFXNWvVUX2167L2jB2kfd6PVmjHnSGBNujInCms+WGmNupq715uzR3TqMBk/GOrqfCPzV2fHYxdUV66ybHUC8s2MDPsX687scax/cnVj75n4DDgC/AsFNKLb5wC5gJ9bk2dFJsY3G2p2yE9huu01uCnVXS2xOrzugH7DNFkMc8IzteFdgI5AAfAH4NqHYltrqLQ74CNtMGGfdgPH8b5ZLnepNrxRVSik34WpdLkoppWqgCV0ppdyEJnSllHITmtCVUspNaEJXSik3oQldKaXchCZ0pZRyE5rQlVLKTfx/hWWZh6OwFU8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#调用\n",
    "epoch = 20\n",
    "Loss = []\n",
    "Accuracy = []\n",
    "for epoch in range(1, epoch+1):\n",
    "    print(\"start_time\",time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())))\n",
    "    loss, acc = train_runner(model, device, trainloader, optimizer, epoch)\n",
    "    Loss.append(loss)\n",
    "    Accuracy.append(acc)\n",
    "    test_runner(model, device, testloader)\n",
    "    print(\"end_time: \",time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())),'\\n')\n",
    "\n",
    "print('Finished Training')\n",
    "plt.subplot(2,1,1)\n",
    "plt.plot(Loss)\n",
    "plt.title('Loss')\n",
    "plt.show()\n",
    "plt.subplot(2,1,2)\n",
    "plt.plot(Accuracy)\n",
    "plt.title('Accuracy')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96e0c005-e5f4-44b0-aa3c-fc7b9adeb72c",
   "metadata": {},
   "source": [
    "## 2.7 保存模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9c3ce7a6-98c8-4d2d-8c1a-bc7b66c6f775",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LeNetRGB(\n",
      "  (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (relu): ReLU()\n",
      "  (maxpool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (maxpool2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (fc1): Linear(in_features=400, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Administrator\\.conda\\envs\\pytorch\\lib\\site-packages\\torch\\serialization.py:360: UserWarning: Couldn't retrieve source code for container of type LeNetRGB. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n"
     ]
    }
   ],
   "source": [
    "print(model)\n",
    "torch.save(model, './models/model-cifar10.pth') #保存模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34e6d0d9-513e-450b-bfe4-288ee5a11c75",
   "metadata": {},
   "source": [
    "## 2.8 模型测试"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b10d2841-d253-451f-8936-9279e1694478",
   "metadata": {},
   "source": [
    "利用刚刚训练的模型进行CIFAR10类型图片的测试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9f12b716-cb14-45d0-bec1-a8d76d202f31",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAR1klEQVR4nO2dy44d13WGd13O6XP6wu5mUySbEkXaimNFyEW+wIaVxIgRKAGCIDACONM8R/IAeRM/gQf2JDYyMCIHUUTBVhKToZRIVJNsNrubfTvXqspAyGz/P8GCHS0Y3zc8C7tq1+U/Bax/77WKrusSAMSj/LwnAAB5ECdAUBAnQFAQJ0BQECdAUGoXfPMfljKV23WtHNcVxQtPpDBJ4xc/2nMG9j2gyWwX7qDufD2S5Z0Z1Hse/5+4a1Zz/DW4CqV9T/X5ZMTdejP9d/9umB3JlxMgKIgTICiIEyAoiBMgKIgTICiIEyAo1kqp6koHO63rPknvX0uWX1op/c7m0uG/apxd4vBX9uK5/p5uw3Nmrw+qIz2sDXu850c14l7ZU734ufhyAgQFcQIEBXECBAVxAgQFcQIEBXECBMVaKaXN/pqdEX08h742hUlRq5CzKfqn5XsO7HOr3A4edz/cyQqxy8jN3cTsHLsXv5NVqQ9YmU/MotEx5xQOzDs8b/Pzb8x1FUnv4lIy5MsJEBTECRAUxAkQFMQJEBTECRAUm62tK5PVdPV0RNLKZRL74rJ4tfrrMVm1uUmqNW2/2j19LtsmcW0qtFeo1yT7dgtwyXz1zu2MTNrV3K1nM31d40o/7NWBPubpIp/mPZtrOdnaTgK+nABBQZwAQUGcAEFBnABBQZwAQUGcAEHxNYRMqrlrdaws8pq3dXFMpllaIimlFWP3lGKx8aLVl+1uSGl2AvTtFqAP6Wwsd8Se9kYfl8s+Th0clXMZUxbGqNRWytxYXINSP9G21cecLfQNadr8C1kaW48aQgC/QSBOgKAgToCgIE6AoCBOgKAgToCg+BpCroiQsEtS6ldsvy61NZPv+/sZg0KnwxvZfVsXj6kqHSuNh+F2aNhaTGKccQdSaf5SbZ2gHjaLtW3M9pI6LWVspdBWSilsioulfj+W5r1qRb2flFJadvr1n5lxnThfpeowpfScLUH5d44vJ0BQECdAUBAnQFAQJ0BQECdAUBAnQFCslTJ00u2xI8EtzO/bscAV5OrEf09lLACXDq8H+ly7W3rc5qo+3/F5PrZ3pG/+fKnvVmnvlimS1auDhhuk78ek1XbVRFgYhbG/rHnk3jnXusJaSOra+piIGr6cAEFBnABBQZwAQUGcAEFBnABBQZwAQbFWypqJdmbbRCvy1663b99u06lzu2PE7gGzA2alWsjY5Q2dzn9pQ4bsTpHNUX4u87G2PY4v9LmW5rmowlQW5ymYUGusj181rpO6s5Z8p293QvF7v1sl4csJEBTECRAUxAkQFMQJEBTECRAUm60dmBL4nVlEvRALm1UWN6XnJcdMWwizeLkUc1wpdX2bcaWvq2j0ufafmFo1Jk09FE+glPWPUhqbROiFycg2vZZfG8zh3L9+rwyqW8DuajuZefRtXdGn9YZdSC/gywkQFMQJEBTECRAUxAkQFMQJEBTECRAUa6UUSZfNb529IXLNqoz983EWjLZFRnU+NjJ2iashNNW3I01nZo6mH0PTqBy7K46k/1PdPS7NtUkLw/djMLEX7zieUkoDcWmtOd7C2EfOautrpfSix6vPlxMgKIgTICiIEyAoiBMgKIgTICiIEyAo1kqZm+r9jen8m8SOitIszXf1XFzZ/Mp0th6KXTWlGbNYmpS9dm3S3LQYMKdLTZu/V66mUmN2xzh3oDIWhqrD49oZ2LpPzoIxO270/XD0KfjznFG2dpKynewBTTAPX06AoCBOgKAgToCgIE6AoCBOgKAgToCgWCvF7R5wmXJZSMoMKkzBMFfYvzBp+Zloez01c186m6J03ZX1/9xKZYp1reSPeT5z7S5kyBZlM5tj0lLsjjEukO8obTyYxlkp4nfVWuOzmD5e6d45627Yq1MT0fPASgH4zQFxAgQFcQIEBXECBAVxAgQFcQIE5TkFvoy94VL24nfXK6VzRauMd9CZtLyMFPpcxi3x/TrM1pM10b06Jf0ALtx2kNJ08za7KRpjD8gNN7a/Td+u0S++U8RdV+HulS001rNXygufqVd9L76cAFFBnABBQZwAQUGcAEFBnABBQZwAQbFWSlUuZKxptK4X7TB/MmO/tMaKKE3bkMbtFFEDXRsSsyvFpcProZ7/dKbHqd0gnelrUpn/VFEf67Njmh0atbIjXJ0rY4n44l89jml3NLndQmZ3j7tZxm5TV+1sPa0kDV9OgKAgToCgIE6AoCBOgKAgToCg2GztVzZ+KGPj6lzGDqa72d+fTm7IMd1yQ8amRT77m1JK3eRCxoaL/H/Pcmiye6suo2zyte1AhppOp2u7tJof0470uUy2s7WbBMx/sVhY7q7ZZV0LE3SZaLXQ3tasssdzWV7Hiy+Kty0oesCXEyAoiBMgKIgTICiIEyAoiBMgKIgTICjWSjmeaAtjde1jGbs5zsd2Fjp5Paqvy9j+dEvGVhaPZOxWka+MM91fk2O6sUm9785lrLli6gQNtO00WeTPN2kvyTGL7rKOLfW4yYm2q2Yn4+zvzsaaj9dl7KLUVtBJs6KPmZQlZeo+2ZX0xtIxx3Q1kPRi+n4dthV8OQGCgjgBgoI4AYKCOAGCgjgBgoI4AYJirZT/uHhbxv7z7C0ZW62Psr+vz/5Fjxnp9PrsXO88GTTawlje+FL293aud57Mf/kLGbt0ps+18XreikgppXrlsYw1Z0+yv0/al+SYgalIszXSj/TWpSsylsTumOmBsWbOtSV1sqOtlKdDbekcL/JWykcneu7NxYGMtYunMjZY1ddW1tquqkf5XVfL5YmeR6PttJTyx+PLCRAUxAkQFMQJEBTECRAUxAkQFMQJEBRrpaiiTyml1Ja6oNVZejX/+2Qix3QX+n+ianSqPM10Wv6/T76WH1Joa2O+onfbjA71/Rjee03Gqmd5uySllPafiOJfm2ZH0FDbNm061eM29c6Icfc/2d/Lcz2P8kTHpnvPZOxkqN+d8+O89fFkpq2ZutNW28Wpnke9rl//styWsbdez9tc46Sf8zsfm4Jt6dv5OZgRAPA5gjgBgoI4AYKCOAGCgjgBgoI4AYJirZSzo5/LWDPVq/3r0Wb29+JU7/jozrVdUiS9i6RIOo1+/PBO9vfy8FCOefPVfRm7/6Eed3BX76qpi/wunZRS+vQkb6Vs734ixzQbZqfIRM/j7OMPZawWuzc603G8qLU105zJUHo201ZQl/L3Y3VD78Qpxvo1bqba/jo90tf2xqv3ZOxPr+Tv1dpA71o62HtFxhR8OQGCgjgBgoI4AYKCOAGCgjgBgmKztZ+89/c6uNRZ0pTyi3yLVi98Hy+nMlbXOuNmEpdpdphfoHxpqTtbT/Y/0LGHOgVZ6umn3Wv6fFuX8y0NzsyC/rf/SC9u/+k/67o+zw71PG7ezF/bxkhnIJ8+MfWbts2CedMiYTnLz7FtTIftuW6TMRyZxf5D/W3669/V9/jqWj5zXJX6ur55S2f6FXw5AYKCOAGCgjgBgoI4AYKCOAGCgjgBgmKtlMO9Yxkbrek0elWoRcqmBk8jaumklMrGdBku9SXsbuUXgZ/saRto77G2e0Zr+r9sbd3ZPTr2t3+Zv+6Xrmpv5tUb+t7vjvX837mjbZa/+m7+mNfezbdpSCml//pHXWfn4a18V/GUUrqzpa2PD0f5Z/PwWA5Jh/t6AXtbaPvob76t6wv9wS39jpyKzgqjoX4uL2/q56LgywkQFMQJEBTECRAUxAkQFMQJEBTECRAUa6WMhjoNXZVa12ofwFJnmm2tmtWhTr0/O9Rl7odFvrXC4RO9u2S8oy2AqtDXXFZ6/vY2z/PnuzLSxzvW3STSt35Hd1f+6pd0On/R5TtHFx++LMf8fqltlm/e1VbEd/5Yz/H+n+ftjZ9/pO2oe+Z+zJe69tBbt7Vd1TT63V+p88+sLPQOmI2hfq8UfDkBgoI4AYKCOAGCgjgBgoI4AYKCOAGCYq2UstKp90GrLYw1EWpanZ5e7XQbgXGlx50bK2X7dt4yOdjQae2m0bdkNDa7Y8ytLApd7OrBg/wxO+M7jfStSnWl5zg3RdkePMwXFPutN/5MjlnZ0RbX2Y9+KmPpdW2lvP5afo5vfEFf9HyYL+SWUkpt0m1DBgs9/8PH2hZpxa4U4y6mqjI+ooAvJ0BQECdAUBAnQFAQJ0BQECdAUBAnQFCslZL0gv70xZt66NtfH2R/31rVByxb/T/RtfkdEymldHp+WcZG9ZPs79/47WM5pl5om2VloIuQFZ1OvafKdJue5C2Md35hdulUuu/G9W09btHqdP7+Qd6Ceelr+v62pe4nMrul348j0428+Pf8HEfr+nhf/j0de/mq6ZXi+sAc62O+/7O8NbaY6fejNa+Hgi8nQFAQJ0BQECdAUBAnQFAQJ0BQbLb2K1+/LWPbG/mMbEopTURycmdNl7+/vaMX2e9e19nJ8UAvom6m+ezZoNYLnotSZ/BcRrY02bim09f2+FF+4L9+cFWOee/uKzK298kjGbt49lDGFsf52OH978sxlyY6Mzx89JGMza7rZ1Yf5e//wak+18/ePZKxP/yWHndNmwBpNrgmY4uVfDv12dEv5ZhBZVqfC/hyAgQFcQIEBXECBAVxAgQFcQIEBXECBMVaKS/f1LlmV67+geh28Okkn4JOKaX3D67L2JY6YErp6qZOo1+5lB/XzHSH58M9fV1pZaxDW/kF7CmldPCxXiC+dz+/CLy6obtGN7d1zZzFVFsAs9Nbepxo13z/qbaxmkLbA9WOadfxWL8HxWHe+jg91BsSjg/1hoof/0SGUllpO3B7Rz+z731vPfv7+khvEnjwTNt3Cr6cAEFBnABBQZwAQUGcAEFBnABBQZwAQbFWymxqSsgXOn3dlfm093Kht26cTvXugcenuuXCvX3debmd5+f//o/uyTEXx7qdwVe/q62I8XVtpdz9pwcy9t4P9rK/v/kX2j768p9syVhnuiuXg7wFkFJK9UbeXlrd1fejaVy3Zv2/X7lyS11+3KVCz+OVTsdm5/pkC9FVPKWUpk/1TqIf3slbMLtfuCHHnLu27gK+nABBQZwAQUGcAEFBnABBQZwAQUGcAEGxVsqk1On8lEw+XGS2C3O2amC6RjfaZmmT3kVy9Di/0+L88adyzHSujzc91uX2K/M3N3uqdzhUKX/MyUO9G2T/jm7vcHqmWx1cnGv7a3qWn0c7Ny00zDUPR3rHx6XL2tJZ28xf2+qavubRUHcOr8yDKSq9U2Q50u/jk+P8O3L0gd4htXlF3w8FX06AoCBOgKAgToCgIE6AoCBOgKAgToCgWCul0tnk1C7MTgDRo2Q606nrxVSn7C/OtYUxm+l5XBzkbYXS9HlZN4XLLt7XvUaOt/Uxh2t6jm9854vZ31fX9U6cdqHv46V1bTls7+gCZanO/08XtbbMKjEmpZQq0zymrvRr1xX53Ruzqb7maaffj9SY+Q/1/Fev6/u4NcjPv0u6cFw1MIXjBHw5AYKCOAGCgjgBgoI4AYKCOAGCgjgBgmKtlMVEF0CaG+vj7DxfHOniQqe825kugNS64kjG7tm8nLcOrty6LcdcLnXK+xuPdDr/7mu6t8n5tk7L1+L/0XS4T1Wn59glfa8aEytF1a3CFAxzG5NaU3Sra/U3oevyO5DKDX0PhfvyfweUodYM1PugUroo8rowtyOV4rocfDkBgoI4AYKCOAGCgjgBgoI4AYJis7WNyYSWZtHz2kZ+0fb6plkYXOgMZOkyhubvpRb1Yzoz9+Hc5un0udb0QvWFyeMtRUuDpcmsulTucqEz7F2rr60SWWpXgyeZxe0O01khqfS7G1Ka++uyza15wRem1YS8x2aShXm/FXw5AYKCOAGCgjgBgoI4AYKCOAGCgjgBgmKtlNak0etap4Yrsdi4bV0LB9OB2OSoO7OwedEKO8IspJ8YC+Pfruk6QfNSL4rvnCsigq2ZR2FaULjaPeNa1xAqxTErY0Uszb134wbm3UliXKOeZUqpNNfsOn23xvpYmhYgiyK/gaMxD7pxOxkEfDkBgoI4AYKCOAGCgjgBgoI4AYKCOAGCUjgrAgA+P/hyAgQFcQIEBXECBAVxAgQFcQIEBXECBOV/AS1hAXgSsRs2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "概率： tensor([[7.6907e-01, 3.3997e-03, 4.8003e-03, 4.2978e-05, 1.2168e-02, 6.8751e-06,\n",
      "         3.2019e-06, 1.6024e-04, 1.2705e-01, 8.3300e-02]],\n",
      "       grad_fn=<SoftmaxBackward>)\n",
      "5\n",
      "预测类别： plane\n"
     ]
    }
   ],
   "source": [
    "from PIL import Image\n",
    "import numpy as np\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "    model = torch.load('./models/model-cifar10.pth') #加载模型\n",
    "    model = model.to(device)\n",
    "    model.eval()    #把模型转为test模式\n",
    "    \n",
    "    #读取要预测的图片\n",
    "    # 读取要预测的图片\n",
    "    img = Image.open(\"./images/test_cifar10.png\").convert('RGB') # 读取图像\n",
    "    #img.show()\n",
    "    plt.imshow(img) # 显示图片\n",
    "    plt.axis('off') # 不显示坐标轴\n",
    "    plt.show()\n",
    "    \n",
    "    # 导入图片，图片扩展后为[1，1，32，32]\n",
    "    trans = transforms.Compose(\n",
    "        [\n",
    "            #将图片尺寸resize到32x32\n",
    "            transforms.Resize((32,32)),\n",
    "            transforms.ToTensor(),\n",
    "            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n",
    "        ])\n",
    "    img = trans(img)\n",
    "    img = img.to(device)\n",
    "    img = img.unsqueeze(0)  #图片扩展多一维,因为输入到保存的模型中是4维的[batch_size,通道,长，宽]，而普通图片只有三维，[通道,长，宽]\n",
    "    \n",
    "    # 预测 \n",
    "    classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')\n",
    "    output = model(img)\n",
    "    prob = F.softmax(output,dim=1) #prob是10个分类的概率\n",
    "    print(\"概率：\",prob)\n",
    "    print(predict.item())\n",
    "    value, predicted = torch.max(output.data, 1)\n",
    "    predict = output.argmax(dim=1)\n",
    "    pred_class = classes[predicted.item()]\n",
    "    print(\"预测类别：\",pred_class)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d63545ed-4970-482a-928c-542aaaced98d",
   "metadata": {},
   "source": [
    "# 三、用LeNet-5训练自己的数据"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afc92632-0719-4eee-8a18-0123a5391363",
   "metadata": {},
   "source": [
    "下面使用LeNet-5网络来训练本地的数据并进行测试。数据集是本地的LED数字0-9，尺寸为28x28单通道，跟MNIST数据集类似。训练集0-9各95张，测试集0~9各40张。图片样例如图所示："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3c121c6-5ca8-4f57-a872-a57cb17042f8",
   "metadata": {},
   "source": [
    "![](./images/lednum.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b18d4ed6-febd-42fa-b882-7ca1c6f68d89",
   "metadata": {},
   "source": [
    "## 3.1 数据预处理 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aad7f71c-ee55-4255-9fe9-86934c4087ef",
   "metadata": {},
   "source": [
    "### 制作图片数据的索引"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7eb91506-7ef2-4be3-a51c-e2c6a2bcf1d8",
   "metadata": {},
   "source": [
    "对于训练集和测试集，要分别制作对应的图片数据索引，即train.txt和test.txt两个文件，每个txt中包含每个图片的目录和对应类别class。示意图如下："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7430607d-051f-45b5-bb89-62de634df43a",
   "metadata": {},
   "source": [
    "![](./images/index.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c075583e-f374-4a18-bde3-d413a7fcc5f0",
   "metadata": {},
   "source": [
    "制作图片数据索引的python脚本程序如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c8ffe39c-1a0a-4117-bb88-0fa1e4af5aff",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "train_txt_path = os.path.join(\"data\", \"LEDNUM\", \"train.txt\")\n",
    "train_dir = os.path.join(\"data\", \"LEDNUM\", \"train_data\")\n",
    "valid_txt_path = os.path.join(\"data\", \"LEDNUM\", \"test.txt\")\n",
    "valid_dir = os.path.join(\"data\", \"LEDNUM\", \"test_data\")\n",
    "\n",
    "def gen_txt(txt_path, img_dir):\n",
    "    f = open(txt_path, 'w')\n",
    "    \n",
    "    for root, s_dirs, _ in os.walk(img_dir, topdown=True):  # 获取 train文件下各文件夹名称\n",
    "        for sub_dir in s_dirs:\n",
    "            i_dir = os.path.join(root, sub_dir)             # 获取各类的文件夹 绝对路径\n",
    "            img_list = os.listdir(i_dir)                    # 获取类别文件夹下所有png图片的路径\n",
    "            for i in range(len(img_list)):\n",
    "                if not img_list[i].endswith('jpg'):         # 若不是png文件，跳过\n",
    "                    continue\n",
    "                label = img_list[i].split('_')[0]\n",
    "                img_path = os.path.join(i_dir, img_list[i])\n",
    "                line = img_path + ' ' + label + '\\n'\n",
    "                f.write(line)\n",
    "    f.close()\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    gen_txt(train_txt_path, train_dir)\n",
    "    gen_txt(valid_txt_path, valid_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba4c80d5-49b3-4320-a735-c2aa08303f57",
   "metadata": {},
   "source": [
    "运行脚本之后就在./data/LEDNUM/目录下生成train.txt和test.txt两个索引文件。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dacc2cd-6dad-4d96-95b1-14dc6d8a60d5",
   "metadata": {},
   "source": [
    "### 构建Dataset子类"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21816f87-9a27-4afb-9777-4e7062ea9364",
   "metadata": {},
   "source": [
    "pytorch 加载自己的数据集，需要写一个继承自torch.utils.data中Dataset类，并修改其中的__init__方法、__getitem__方法、__len__方法。默认加载的都是图片，__init__的目的是得到一个包含数据和标签的list，每个元素能找到图片位置和其对应标签。然后用__getitem__方法得到每个元素的图像像素矩阵和标签，返回img和label。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "26c4a836-480f-4d34-b678-19f28c5c01c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "from torch.utils.data import Dataset\n",
    "\n",
    "class MyDataset(Dataset):\n",
    "    def __init__(self, txt_path, transform = None, target_transform = None):\n",
    "        fh = open(txt_path, 'r')\n",
    "        imgs = []\n",
    "        for line in fh:\n",
    "            line = line.rstrip()\n",
    "            words = line.split()\n",
    "            imgs.append((words[0], int(words[1])))\n",
    "            self.imgs = imgs \n",
    "            self.transform = transform\n",
    "            self.target_transform = target_transform\n",
    "    def __getitem__(self, index):\n",
    "        fn, label = self.imgs[index]\n",
    "        #img = Image.open(fn).convert('RGB') \n",
    "        img = Image.open(fn)\n",
    "        if self.transform is not None:\n",
    "            img = self.transform(img) \n",
    "        return img, label\n",
    "    def __len__(self):\n",
    "        return len(self.imgs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7445765-1da5-4742-9391-a8b8e3571b5e",
   "metadata": {},
   "source": [
    "getitem是核心函数。self.imgs是一个list，self.imgs[index]是一个str，包含图片路径，图片标签，这些信息是从上面生成的txt文件中读取；利用Image.open对图片进行读取，注意这里的img是单通道还是三通道的；self.transform(img)对图片进行处理，这个transform里边可以实现减均值、除标准差、随机裁剪、旋转、翻转、放射变换等操作。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c0a6ad3-c5ba-416b-a5a0-fa0686f15885",
   "metadata": {},
   "source": [
    "当Mydataset构建好，剩下的操作就交给DataLoder，在DataLoder中，会触发Mydataset中的getiterm函数读取一张图片的数据和标签，并拼接成一个batch返回，作为模型真正的输入。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "36c1f813-e1c1-413b-b141-a86ad3b01e58",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipline_train = transforms.Compose([\n",
    "    #随机旋转图片\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    #将图片尺寸resize到32x32\n",
    "    transforms.Resize((32,32)),\n",
    "    #将图片转化为Tensor格式\n",
    "    transforms.ToTensor(),\n",
    "    #正则化(当模型出现过拟合的情况时，用来降低模型的复杂度)\n",
    "    transforms.Normalize((0.1307,),(0.3081,))    \n",
    "])\n",
    "pipline_test = transforms.Compose([\n",
    "    #将图片尺寸resize到32x32\n",
    "    transforms.Resize((32,32)),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize((0.1307,),(0.3081,))\n",
    "])\n",
    "train_data = MyDataset('./data/LEDNUM/train.txt', transform=pipline_train)\n",
    "test_data = MyDataset('./data/LEDNUM/test.txt', transform=pipline_test)\n",
    "\n",
    "#train_data 和test_data包含多有的训练与测试数据，调用DataLoader批量加载\n",
    "trainloader = torch.utils.data.DataLoader(dataset=train_data, batch_size=8, shuffle=True)\n",
    "testloader = torch.utils.data.DataLoader(dataset=test_data, batch_size=4, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "695d40e7-78ad-4f7c-8a80-44240b46c496",
   "metadata": {},
   "source": [
    "## 3.2 搭建LeNet-5神经网络结构"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a966bbd6-9ae0-4b07-ac8a-f64cf6ac5428",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LeNet(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(LeNet, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 6, 5) \n",
    "        self.relu = nn.ReLU()\n",
    "        self.maxpool1 = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        self.maxpool2 = nn.MaxPool2d(2, 2)\n",
    "        self.fc1 = nn.Linear(16*5*5, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    " \n",
    "    def forward(self, x):\n",
    "        x = self.conv1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.maxpool1(x)\n",
    "        x = self.conv2(x)\n",
    "        x = self.maxpool2(x)\n",
    "        x = x.view(-1, 16*5*5)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        output = F.log_softmax(x, dim=1)\n",
    "        return output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d006592e-4447-4729-a58a-6e61ed0a52a6",
   "metadata": {},
   "source": [
    "## 3.3 将定义好的网络结构搭载到GPU/CPU，并定义优化器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d94a236c-d6d6-4f7e-8146-b01a62904208",
   "metadata": {},
   "outputs": [],
   "source": [
    "#创建模型，部署gpu\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "model = LeNet().to(device)\n",
    "#定义优化器\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6accdf41-017e-47e0-8437-e9ef990bbfba",
   "metadata": {},
   "source": [
    "### 3.4 定义训练函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "8c9f793d-3f6e-461b-82b3-c6108b1d32f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_runner(model, device, trainloader, optimizer, epoch):\n",
    "    #训练模型, 启用 BatchNormalization 和 Dropout, 将BatchNormalization和Dropout置为True\n",
    "    model.train()\n",
    "    total = 0\n",
    "    correct =0.0\n",
    "    \n",
    "    #enumerate迭代已加载的数据集,同时获取数据和数据下标\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        inputs, labels = data\n",
    "        #把模型部署到device上\n",
    "        inputs, labels = inputs.to(device), labels.to(device)\n",
    "        #初始化梯度\n",
    "        optimizer.zero_grad()\n",
    "        #保存训练结果\n",
    "        outputs = model(inputs)\n",
    "        #计算损失和\n",
    "        #多分类情况通常使用cross_entropy(交叉熵损失函数), 而对于二分类问题, 通常使用sigmod\n",
    "        loss = F.cross_entropy(outputs, labels)\n",
    "        #获取最大概率的预测结果\n",
    "        #dim=1表示返回每一行的最大值对应的列下标\n",
    "        predict = outputs.argmax(dim=1)\n",
    "        total += labels.size(0)\n",
    "        correct += (predict == labels).sum().item()\n",
    "        #反向传播\n",
    "        loss.backward()\n",
    "        #更新参数\n",
    "        optimizer.step()\n",
    "        if i % 100 == 0:\n",
    "            #loss.item()表示当前loss的数值\n",
    "            print(\"Train Epoch{} \\t Loss: {:.6f}, accuracy: {:.6f}%\".format(epoch, loss.item(), 100*(correct/total)))\n",
    "            Loss.append(loss.item())\n",
    "            Accuracy.append(correct/total)\n",
    "    return loss.item(), correct/total"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75074ca9-a613-4ace-8a13-5ed991c51751",
   "metadata": {},
   "source": [
    "## 3.5 定义测试函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "19f6ef40-ea97-4f22-a517-4706fa3da297",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_runner(model, device, testloader):\n",
    "    #模型验证, 必须要写, 否则只要有输入数据, 即使不训练, 它也会改变权值\n",
    "    #因为调用eval()将不启用 BatchNormalization 和 Dropout, BatchNormalization和Dropout置为False\n",
    "    model.eval()\n",
    "    #统计模型正确率, 设置初始值\n",
    "    correct = 0.0\n",
    "    test_loss = 0.0\n",
    "    total = 0\n",
    "    #torch.no_grad将不会计算梯度, 也不会进行反向传播\n",
    "    with torch.no_grad():\n",
    "        for data, label in testloader:\n",
    "            data, label = data.to(device), label.to(device)\n",
    "            output = model(data)\n",
    "            test_loss += F.cross_entropy(output, label).item()\n",
    "            predict = output.argmax(dim=1)\n",
    "            #计算正确数量\n",
    "            total += label.size(0)\n",
    "            correct += (predict == label).sum().item()\n",
    "        #计算损失值\n",
    "        print(\"test_avarage_loss: {:.6f}, accuracy: {:.6f}%\".format(test_loss/total, 100*(correct/total)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "300ad714-e233-4840-99f0-b53a76d53098",
   "metadata": {},
   "source": [
    "## 3.6 运行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "de367098-3488-4bf2-a91b-53fb38ee53ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "start_time 2021-11-27 22:29:46\n",
      "Train Epoch1 \t Loss: 2.381566, accuracy: 12.500000%\n",
      "Train Epoch1 \t Loss: 0.016064, accuracy: 75.123762%\n",
      "test_avarage_loss: 0.013390, accuracy: 98.250000%\n",
      "end_time:  2021-11-27 22:29:47 \n",
      "\n",
      "start_time 2021-11-27 22:29:47\n",
      "Train Epoch2 \t Loss: 0.005825, accuracy: 100.000000%\n",
      "Train Epoch2 \t Loss: 0.001058, accuracy: 99.752475%\n",
      "test_avarage_loss: 0.026550, accuracy: 96.250000%\n",
      "end_time:  2021-11-27 22:29:48 \n",
      "\n",
      "start_time 2021-11-27 22:29:48\n",
      "Train Epoch3 \t Loss: 0.149143, accuracy: 87.500000%\n",
      "Train Epoch3 \t Loss: 0.001346, accuracy: 99.628713%\n",
      "test_avarage_loss: 0.008281, accuracy: 98.000000%\n",
      "end_time:  2021-11-27 22:29:50 \n",
      "\n",
      "start_time 2021-11-27 22:29:50\n",
      "Train Epoch4 \t Loss: 0.002901, accuracy: 100.000000%\n",
      "Train Epoch4 \t Loss: 0.001207, accuracy: 99.504950%\n",
      "test_avarage_loss: 0.000260, accuracy: 100.000000%\n",
      "end_time:  2021-11-27 22:29:51 \n",
      "\n",
      "start_time 2021-11-27 22:29:51\n",
      "Train Epoch5 \t Loss: 0.000148, accuracy: 100.000000%\n",
      "Train Epoch5 \t Loss: 0.005780, accuracy: 99.752475%\n",
      "test_avarage_loss: 0.001154, accuracy: 100.000000%\n",
      "end_time:  2021-11-27 22:29:53 \n",
      "\n",
      "Finished Training\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAACSCAYAAABsboAjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAASHElEQVR4nO3df2zcd33H8ef77vwjdpzYZztxfttJwVko/ZGmdlomNBUYGUN00vYHFVRsMPUf2GCrxOjKEJumrRNTAY2xqYKuSHRFEzCBEGwU6ASINIkTWlqatiSOkyZNYsd2EieO4x/33h/f7zlnx3bcxOfvD78e0sn3/eHz6873ffnrz33ve+buiIhIfGWiDiAiInNTUYuIxJyKWkQk5lTUIiIxp6IWEYk5FbWISMypqEVEYk5FLYlmZj1m9s6oc4iUk4paRCTmVNSSOmZWZWZfMLPXw8sXzKwqXNZkZt8zs7NmNmBmPzOzTLjsr8zshJkNmdkrZvaOaO+JSCAXdQCRMngY2AncBjjwHeDTwN8ADwLHgeZw3Z2Am1k78DHgTnd/3cxagezixhaZmfaoJY0+APydu/e6ex/wt8D94bIxYA2wyd3H3P1nHpzwZgKoAraZWYW797j74UjSi0yjopY0WgscLZk+Gs4D+BxwCPihmXWb2acA3P0Q8Angs0CvmX3DzNYiEgMqakmj14FNJdMbw3m4+5C7P+jum4H3AX9ZHIt29/90998Ov9eBf1rc2CIzU1FLGlSYWXXxAjwFfNrMms2sCfgM8HUAM3uvmd1kZgacIxjyKJhZu5ndE77oOAJcAgrR3B2RqVTUkgbfJyjW4qUa6AJ+BbwAHAD+Plz3TcCPgAvAbuDL7v4Mwfj0I8AZ4BSwCnho8e6CyOxMHxwgIhJv2qMWEYk5FbWISMypqEVEYk5FLSIScypqEZGYK8u5Ppqamry1tbUcNy0ikkr79+8/4+7NMy0rS1G3trbS1dVVjpsWEUklMzs62zINfYiIxFxsinqi4Dzb3c+rp4eijiIiEiuxKeqCOx9+Yh9PPjvr3r+IyJIUm6KuyGa4Y1MDe44MRB1FRCRWYlPUAB2teV45PcTZ4dGoo4iIxEa8irotjzvs6xmMOoqISGzEqqhv3VBPZS7D3iP9UUcREYmNWBV1dUWW2zbUa5xaRKRErIoaYGdbnhdPnOPC5fGoo4iIxELsirqjrZGCw/6jGqcWEYEYFvX2TfXkMsaebo1Ti4hADIu6pjLHW9evZK/GqUVEgBgWNQSH6T1//CwjYxNRRxERidw1i9rMNpjZM2b2kpn92sw+Xu5QnW15xiacA8c0Ti0iMp896nHgQXffBuwEPmpm28oZakdrHjM0/CEiwjyK2t1PuvuB8PoQcBBYV85QK6or2LZmBXu6VdQiIm9ojNrMWoHbgT1lSVOioy3PgWODjI4Xyv2jRERibd5FbWbLgW8Bn3D38zMsf8DMusysq6+v74aDdbY1cnm8wAsnzt7wbYmIJNm8itrMKghK+kl3//ZM67j7Y+6+w913NDfP+LFfb0hHWx6AZzX8ISJL3HyO+jDgq8BBd3+0/JEC+dpK3rx6uV5QFJElbz571G8D7gfuMbPnwst7ypwLCPaq9x8dZHxC49QisnTN56iPn7u7ufst7n5bePn+YoTraGvkwuVxXjp51ZC4iMiSEct3JhZ1huPUGv4QkaUs1kW9ekU1rY01Oj+1iCxpsS5qCMap9/UMUCh41FFERCIR+6LubGvk7PAYr/YORR1FRCQSsS/q4vHUeju5iCxVsS/q9Q3LWLuyWi8oisiSFfuiNjM6Nzey58gA7hqnFpGlJ/ZFDcHwx5kLl+k+czHqKCIiiy4RRa3jqUVkKUtEUbc11dK0vEpFLSJLUiKK2szobMuzp7tf49QisuQkoqgBOjfnef3cCMcHL0UdRURkUSWmqDs0Ti0iS1RiivrNq+qor6lgz5H+qKOIiCyqxBR1JmPc2ZrXHrWILDmJKWoIDtPr6R/m9PmRqKOIiCyaRBX15Hk/tFctIktIoop625oVLK/KsVfj1CKyhCSqqHPZDHdsatCZ9ERkSUlUUUNwPPVvei/Qf+Fy1FFERBZF8oo6HKfe1zMYcRIRkcWRuKJ+67p6qisyOp5aRJaMxBV1ZS7D9o0NOp5aRJaMxBU1BIfpvXTyPOdHxqKOIiJSdoktanfo6tFetYikXyKLevvGBiqypje+iMiSkMiirq7Icuv6eh1PLSJLQiKLGoLhjxdPnOPi5fGoo4iIlFVii7pzcyPjBeeXx85GHUVEpKyuWdRm9riZ9ZrZi4sRaL7u2NRANmM6nlpEUm8+e9RPALvKnOMNW16V4+a1K/SCooik3jWL2t1/CsSyDTva8jz32llGxiaijiIiUjaJHaMG6GhrZHS8wPOvnY06iohI2SxYUZvZA2bWZWZdfX19C3Wzc+pozWOmD7wVkXRbsKJ298fcfYe772hubl6om53TypoK2lfXsVfvUBSRFEv00AcEpz3df3SQsYlC1FFERMpiPofnPQXsBtrN7LiZfaT8seavc3Mjw6MTvHjiXNRRRETKInetFdz9vsUIcr3ubA0+SGDvkQFu39gQcRoRkYWX+KGP5roqNjfX6nhqEUmtxBc1QGdbI/t6BpgoeNRRREQWXEqKOs/QyDgHT56POoqIyIJLRVF3tF0ZpxYRSZtUFPXa+mVsyC9TUYtIKqWiqAE6WhvZ2zOAu8apRSRdUlPUnZvzDFwc5VDvhaijiIgsqPQUdThOrcP0RCRtUlPUG/M1rF5RpaIWkdRJTVGbGZ1tjew90q9xahFJldQUNQSH6Z0+f5ljA8NRRxERWTCpKurJcepuDX+ISHqkqqhvWrWcfG2lxqlFJFVSVdRmRkdrXp9MLiKpkqqihuB46uODlzhx9lLUUUREFkTqirp43o99Gv4QkZRIXVFvbVlBXXVOwx8ikhqpK+pspjhOrT1qEUmH1BU1BMMf3X0X6Ru6HHUUEZEbltqiBp2fWkTSIZVFffO6ldRUZtmrcWoRSYFUFnVFNsMdmxo0Ti0iqZDKogboaM3zyukhzg6PRh1FROSGpLaoOzc34g77egajjiIickNSW9S3rF9JZS7Dnm6NU4tIsqW2qKsrsty+oZ69PRqnTppCwbk8PhF1DJHYyEUdoJw62/J86ZlDXLg8zvKqVN/VxDp3aYxXTg3x8qnzvHxqiJdPnufV0xcYGZvg1g313L2lkbs2N7J9UwPVFdmo44pEItXt1dHWSOEnh+jqGeB32ldFHWdJG5socOTMRQ6ePB8Wc1DKr58bmVxn5bIK2lvq+MPt66iuzLKne4Av/99h/uUnh6jMZbhjYwN3bWnk7i2N3LK+nspcav8hFJki1UW9fVM9uYyx94iKerG4O31DlzkYFvErp4Y4eGqIw70XGJ0oAJDLGDetWs6dbXm2tqxg65o6trbU0bKiGjObcntDI2Ps6xngF4f62d3dz+d/9CqPPg01lVl2tOa5Oyzut6xdSTZjM0USSbx5FbWZ7QK+CGSBr7j7I2VNtUBqKnO8df1KvUOxDC6NTjAwPErv+RFePT3EwZNDk0MYg8Njk+u1rKimvaWOt7+5id9qWUF7Sx1bmpfPe2+4rrqCe7au5p6tqwEYvDjKniP97D7czy8O9/PID14O18vR2RaU9l1bGmlfXUdGxS0pcc2iNrMs8K/Au4DjwD4z+667v1TucAuhoy3P4z8/wvmRMZZX5rTxzmBsosDg8CgDF4PL4MUxBoZHGbxYMm+4uGyUgeFRRsYKU25jWUWW9pY63v2WFra21NHesoKtLXU01FYuaNaG2kp23byGXTevAaB3aIRnuwfYffgMuw/386ODpwHI11Zy1+ZGdoZ73Jubaq/aWxdJCrvWJ3ab2V3AZ9393eH0QwDu/o+zfc+OHTu8q6trIXNet2de7uVPntg3OZ3LGBXZDBVZozKXCa8H0xXZTMm8cLq4PBd+T/bK9+SyRqHgOOAOhfCxLLhPTheX+WzzZlgfIGtGxiBjhpVcz2SCT7IpLg+WhcszhhXXs+A2isvN4MLl8cniLS3foZHxWR+/uuoc+dpKGmoqaaytpKG2cnI6X1tBY20VN61azsZ8TSz+CJ44eync2w6K+2Q4Br6qroq7tzTS2lSLMTXn9P6e6V5ctc60GcXfr09Og1Myr/T3XjK/uPk5wQwvua3pP9ssTF6cJvi9Wsl6xfs2OT/8Zptheen8me6TTfs5pY9N6bKZHp+FUtwuCsWvBWci/Fpwguul0wXHPVzHCec7EwWmzncnY0Z2crsKt5mMTdm+gvkWzi9ul8H6V68T/Bf/wZ2bruu+mtl+d98x47J5FPUfAbvc/U/D6fuBTnf/2LT1HgAeANi4ceMdR48eva6wC22i4Dy19xjnLo0xNlEIL87oeGHq9ESBsfFp08XLuDM2USiZ54yNFxgv+OQGkQmvFEsxE25UZlc2sNJlhMumr1/cEq48QZ1C+CQrTH/Shk/Q6U9mL3kCl/56q3KZKWV7pXDDeTWVNITl21BbQf2yykS/YOfuHO0fZnd3MEyy+/AZzlyI7p2qxeeKlT43uNK0pfOK685U/BTnh/OuLC+u61P+CKRVsVgzNq1MS+ZNWSdDWM7Bg1vcbibCbWgiLPXitjQ5XfJHwcP1C7M8ts11Vex7+J3XdX/mKuoFezHR3R8DHoNgj3qhbvdGZTN23X/h0qB0Tz6XTW7pXg8zo7WpltamWu7r2HjVnipcKbfJ6Rna7ep1pi/3KeVa3DMt/UMdtdL7Pr3Qg+vFZT5leq5l02+nXPdytj3aKBUfzyt79Fd2nsphPkV9AthQMr0+nCcJYMU99rJtRslRfCyusdZiRFl0M9/3dN7XxbDY29V8drH2AW8yszYzqwTeD3y3vLFERKTomnvU7j5uZh8D/pfg8LzH3f3XZU8mIiLAPF5MvK4bNesDrvfVxCbgzALGKackZYVk5U1SVkhW3iRlhWTlvZGsm9y9eaYFZSnqG2FmXbO98hk3ScoKycqbpKyQrLxJygrJyluurEvrMAARkQRSUYuIxFwci/qxqAO8AUnKCsnKm6SskKy8ScoKycpblqyxG6MWEZGp4rhHLSIiJWJT1Ga2y8xeMbNDZvapqPPMxcw2mNkzZvaSmf3azD4edaZrMbOsmf3SzL4XdZZrMbN6M/ummb1sZgfDE4PFkpn9RfgceNHMnjKz6qgzlTKzx82s18xeLJmXN7Onzew34deGKDMWzZL1c+Hz4Fdm9t9mVh9hxClmyluy7EEzczNrWoifFYuiLjmV6u8B24D7zGxbtKnmNA486O7bgJ3AR2OeF+DjwMGoQ8zTF4H/cfetwK3ENLeZrQP+HNjh7jcTvCHs/dGmusoTwK5p8z4F/Njd3wT8OJyOgye4OuvTwM3ufgvwKvDQYoeawxNcnRcz2wD8LnBsoX5QLIoa6AAOuXu3u48C3wDujTjTrNz9pLsfCK8PERTJumhTzc7M1gO/D3wl6izXYmYrgbcDXwVw91F3PxtpqLnlgGVmlgNqgNcjzjOFu/8UmP7JGfcCXwuvfw34g8XMNJuZsrr7D929eB7eZwnONRQLszy2AJ8HPsnV5/O6bnEp6nXAayXTx4lx8ZUys1bgdmBPxFHm8gWCJ07hGuvFQRvQB/xHOFTzFTOrjTrUTNz9BPDPBHtOJ4Fz7v7DaFPNy2p3PxlePwWsjjLMG/Bh4AdRh5iLmd0LnHD35xfyduNS1IlkZsuBbwGfcPfzUeeZiZm9F+h19/1RZ5mnHLAd+Dd3vx24SHz+NZ8iHNu9l+CPy1qg1sw+GG2qN8aDw75if+iXmT1MMOT4ZNRZZmNmNcBfA59Z6NuOS1En7lSqZlZBUNJPuvu3o84zh7cB7zOzHoIhpXvM7OvRRprTceC4uxf/Q/kmQXHH0TuBI+7e5+5jwLeBuyPONB+nzWwNQPi1N+I8czKzPwbeC3zA43088RaCP9rPh9vbeuCAmbXc6A3HpagTdSpVC84E/1XgoLs/GnWeubj7Q+6+3t1bCR7Xn7h7bPf63P0U8JqZtYez3gHE9fM5jwE7zawmfE68g5i+8DnNd4EPhdc/BHwnwixzCj9Y+5PA+9x9OOo8c3H3F9x9lbu3htvbcWB7+Jy+IbEo6vDFguKpVA8C/xXzU6m+DbifYO/0ufDynqhDpcifAU+a2a+A24B/iDbOzMK9/m8CB4AXCLanWL2LzsyeAnYD7WZ23Mw+AjwCvMvMfkPwX8EjUWYsmiXrl4A64OlwO/v3SEOWmCVveX5WvP+TEBGRWOxRi4jI7FTUIiIxp6IWEYk5FbWISMypqEVEYk5FLSIScypqEZGYU1GLiMTc/wMfNK9i//jo0QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAACSCAYAAABLwAHLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAYqUlEQVR4nO3de5hU9Z3n8fenqptLA0Jza7AbaBQEUbyQjmJ0jIk3VBRnZ3ZHM0ZMDDw7G2ecWXcmZpM1E+PkcZKZTbJPnImOis6MA5sxToINXlgvMYk3Gi+NNKJcBBoaaGiQO3RXffePOtVUN30poLpP9anv63nqqXN+55w6366u+tSp3zl1jswM55xz0RULuwDnnHM9y4PeOecizoPeOecizoPeOecizoPeOecizoPeOecizoPeOecizoPeRYqkVyXtltQ/7Fqcyxce9C4yJFUCvwcYcFMvrreot9bl3MnwoHdRcjvwJvAEMDfdKGmcpGckNUraJemnGdPmSVotaZ+kOkkzgnaTNCljvickPRAMXyGpXtI3JG0DFkgqlVQdrGN3MFyRsfxwSQskbQ2m/zJo/0DSjRnzFUvaKenCnnqSXOHxoHdRcjvwVHC7VlKZpDhQDWwEKoFyYBGApP8M/HWw3GmkvgXsynJdY4DhwARgPqn30oJgfDxwCPhpxvz/ApQA5wCjgR8F7f8M3JYx3/VAg5m9m2UdznVLfq4bFwWSLgNeAcaa2U5JHwIPk9rCXxy0t7Rb5gVgqZn9pIPHM2Cyma0Nxp8A6s3s25KuAF4ETjOzw53UcwHwipmVShoLbAFGmNnudvOdDqwBys1sr6SngbfN7Acn+VQ4dxzfondRMRd40cx2BuP/FrSNAza2D/nAOGDdSa6vMTPkJZVIeljSRkl7gdeAYcE3inFAU/uQBzCzrcDvgD+QNAy4jtQ3EudyxnciuT5P0kDgvwDxoM8coD8wDNgOjJdU1EHYbwbO7ORhD5LqakkbA9RnjLf/KnwPMAW42My2BVv07wIK1jNc0jAz29PBup4Evkbq/fiGmW3ppCbnTopv0bsouBlIANOAC4Lb2cBvgmkNwIOSBkkaIOnSYLlHgf8h6TNKmSRpQjDtPeBLkuKSZgGf76aGIaT65fdIGg58Jz3BzBqA54B/CHbaFku6PGPZXwIzgLtJ9dk7l1Me9C4K5gILzGyTmW1L30jtDL0VuBGYBGwitVX+RwBm9u/A35Dq5tlHKnCHB495d7DcHuCPg2ld+TEwENhJar/A8+2mfxloBj4EdgB/np5gZoeAXwATgWey/7Ody47vjHUuD0i6DzjLzG7rdmbnTpD30TsXsqCr505SW/3O5Zx33TgXIknzSO2sfc7MXgu7HhdN3nXjnHMR51v0zjkXcR70zjkXcXm3M3bkyJFWWVkZdhnOOdenrFixYqeZjepoWrdBL+lxYDaww8zO7WC6gJ+QOhnTQeAOM3snmDYX+HYw6wNm9mR366usrKSmpqa72ZxzzmWQtLGzadl03TwBzOpi+nXA5OA2H/jHYKXpXwdeDFwEfEdSaXYlO+ecy5Vut+jN7LXggg6dmQP8s6UO33lT0rDgbH1XAMvMrAlA0jJSHxgLT7lq16lNuw6yrnE/sZiIS8RiEJeIx9TaFo+JWHAfj9E6fKwtYzj9GDEhhHRsXRKtbWptE0pPy5zZdcvMSCSNlqSRtNR9IhHcJ42EpceTrfMlktZmOD0tkTTMoCguiuMxiuMx+sVjFBfp2HA8RnFcFBcdG4/H8u9/ZsFz0ZIwmpNJWhJGSyJJczK4D56TloRxNNF2eiKZRDq1131qOOM+JtJPU/qgRWtXb2Zb64GNGTNZMNJ++XhMDB1YnOunMCd99OWkjgNOqw/aOms/jqT5pL4NMH78+ByUVJheX7uTrz65nMPNybBLOU76wyDzgwBg6MBi/uSKSdx+yQSK4/l7bMDra3fyty+sYXPTQczsuDdxZhuW+SZvO2+Hb3AjFeLJ8A91jomMD4XggyDzg6EoFXjGsb/HLD2cEXLt/tb0tPTfmzmenjeRsNbwbhPqefC89JYLxg3jl1+/tPsZT1Be7Iw1s0eARwCqqqoK57+aQ6+vS4X8hOGDeOD3zyUmSCQhEWwdprcIk8HWXqqNNm2tw23aaG07Fk7WNuAyQqvNmx9aE63DaUBt/R6+V13HU29t5H/NnsYXpozuvSctC5/sPMD3l67mxbrtVJQO5PrpY9p8s0l/eKUda28/T3q6Wr/9kDFvUezYVmU8dmy8KCbi8RhxZbTFj02LKT0eO+4xBLQkjeaW1NZtc0uS5kSSo8FWcHMiGG9pN55I0tzSbjxxbPmWpHXywa2M9nbPQTBOB8ulx2MS/YpEUSzW+k2kKCaK4jGK0/dxHWuLHz9vcVGM4ta21POSPO413vnrPvO9cXzbseGkZX6LPfZ/bP86ODauzH95u+WOzTNycM9c6jgXQb+F1Pm20yqCti2kum8y21/NwfpcO2+s28VXn1jO+OElPDXv4h57sfQEM+PlD3fwwJLVfGXBcq6YMopv3zCNSaMHh1rX3sPNPPTyWh7/3QaK4zH+8top3HnZRAYUx0Oty7mTkYugXwzcJWkRqR2vn5pZQ3D1nu9n7IC9BvhmDtbnMry5PhXy40pL+Ld5M/tUyENqK+bKs8v4vcmjePL1T/g/L33MrB+/xu2XVHL3lZMZWpL7/squJJLGz2s28/cvrmHXgaP84YwK/vLaKYw+bUCv1uFcLmVzeOVCUlvmIyXVkzqSphjAzH4GLCV1aOVaUodXfiWY1iTpe8Dy4KHuT++Ydbnx5vpdfGXBcipKB/bJkM/UryjGvMvP4PdnlPP3L37Egtc38B/v1nPPNVO45bPjKOqF/vs31u3i/uo6VjfspWpCKQvuuIjpFUN7fL3O9bS8O9dNVVWV+XH03Xtr/S7uWLCc8tKBLJw3k1FD+m7Id2TV1k+5/9k63trQxNQxQ7hv9jQ+N2lkj6xr066DfH/pap5ftY3yYQP55vVTuWH6WD9qyPUpklaYWVWH0zzo+563NzRxx4K3GTt0AAvnz2T0kGh2K5gZz3+wjb9Zupr63Ye4ZloZ37rhbCaMGJSTx99/pIWHXlnLY7/ZQDwm/tsVZzLv8jO8H971SR70EVIoIZ/pcHOCx367gYdeWUtLwvjqZRO564uTGNz/5HYxJZLGL1bU84MX1rBz/xH+04xyvjFrKmXeD+/6MA/6iFj+SRNzH3+bMUMHsGjezILbQbh972H+9vkPeeadLYwc3J+/unYKf/iZCmIn8COftzc08d1nV7Fq615mjB/GfTeewwXjhvVc0c71Eg/6CKgJQr6sQEM+03ub9/DdZ1fx7qY9TC8fyndunEZV5fAul9ncdJAHn/uQJSsbGDt0APdeN5Wbzj/d++FdZHjQ93GtIX/aABbNL+yQTzMzfvXeVh587kO27T3Mjeefzr3XTaV82MA28x040sI/vLqWf/rNBmKCP/n8JOZffgYD+3k/vIuWroI+L34Z6zq3YuOxkF/oId9KEjdfWM4155Txs1+v5+Ffr2NZ3TbmX34m//XzZzCgKM4v3kn1wzfuO8LNF5zON66bytihA7t/cOcixrfo89iKjbuZ+/jbjBrSn0XzZ/rOwi7U7051zVTXprpmRgzuxwdb9nLBuGHcd+M0Zoz3E6e6aPMt+j4oHfIjB/dj4TwP+e5UlJbw0y/NYO7nmnhgyWqaDhzhR390PnPOLz+hnbXORZEHfR56Z1Mq5EcM7sfC+TMZM9RDPlufrRzOr3rg7H/O9WX5e17YAvXupt3MfSwV8ovmz/Q+ZefcKfOgzyPvbd7D7Y+9TemgVHeNh7xzLhc86PPE+5v38OXH3qJ0UGpL/vRhHvLOudzwoM8D72/ew22PvcWwkmIWesg753LMgz5ktfXHQn7R/EuO+8GPc86dKg/6EK2s/5TbHn2LoQOLWThvpoe8c65H+OGVvexwc4K1O/ZTt3UvDyyp47SBxSyaP5OK0pKwS3PORVRWQS9pFvATIA48amYPtpv+I+ALwWgJMNrMhgXTEsDKYNomM7spB3XnvaMtST7ZdYA12/bx0fb0bT8bdx0gfVH7CSNK+Nc7L/aQd871qGwuJRgHHgKuBuqB5ZIWm1ldeh4z+4uM+f8UuDDjIQ6Z2QU5qzjPJJLGpqaDrNm2j4+372NNEOrrGw/QEiR6TFA5chBTxwzhxvNPZ0rZEKaMGUzliEG9cok851xhy2aL/iJgrZmtBwguAj4HqOtk/ltJXVc2UsyMLXsO8fH2/akw35YK9bU79nOkJdk637jhA5lSNoQrzy5jStkQziobwhmjBvlVi5xzockm6MuBzRnj9cDFHc0oaQIwEXg5o3mApBqgBXjQzH55cqWGo373Qe75+fus2rqX/UdaWtvHnDaAs8YM4ZIzRnDWmCFMKRvCpNGDGXSSVz1yzrmekutUugV42swSGW0TzGyLpDOAlyWtNLN1mQtJmg/MBxg/fnyOSzo1C373Ce9u2sOtF41rDfTJo4cwtKQ47NKccy4r2QT9FmBcxnhF0NaRW4CvZzaY2Zbgfr2kV0n1369rN88jwCOQOk1xNoX3hmTSWLqygcvPGsV355wbdjnOOXdSstkTuByYLGmipH6kwnxx+5kkTQVKgTcy2kol9Q+GRwKX0nnfft55Z9NuGj49zOzzxoZdinPOnbRut+jNrEXSXcALpA6vfNzMVkm6H6gxs3To3wIssrZXMjkbeFhSktSHyoOZR+vku+raBvoVxbhqWlnYpTjn3EnLqo/ezJYCS9u13ddu/K87WO51YPop1BeaRNBt84UpoxjsO1idc32YH8TdiZpPmtix7wizzzs97FKcc+6UeNB3orq2gQHFMb44dXTYpTjn3CnxoO9ASyLJcx80cOXUMj8u3jnX53nQd+DtDU3s3H+UG/xoG+dcBHjQd+DZ2gZK+sX5whTvtnHO9X0e9O20JJI8/0EDV51dxsB+fn4a51zf50HfzuvrdrH7YLN32zjnIsODvp0ltQ0M7l/E588aFXYpzjmXEx70GY62JHl+1TaunlbmpxV2zkWGB32G363byaeHmv3cNs65SPGgz1D9fgNDBhRx2eSRYZfinHM540EfONKS4MW6bVx7zhj6F3m3jXMuOjzoA7/5aCf7Drf40TbOucjxoA9U125lWEkxl03ybhvnXLR40AOHmxMsq9vOrHPGUBz3p8Q5Fy2easCraxo5cDTh3TbOuUjyoAeWrGxg+KB+XHLGiLBLcc65nMsq6CXNkrRG0lpJ93Yw/Q5JjZLeC25fy5g2V9LHwW1uLovPhUNHE7y0ejuzzh1DkXfbOOciqNuTrUuKAw8BVwP1wHJJizu49uv/NbO72i07HPgOUAUYsCJYdndOqs+BV9bs4ODRhP9IyjkXWdlswl4ErDWz9WZ2FFgEzMny8a8FlplZUxDuy4BZJ1dqz6iu3crIwf25eKJ32zjnoimboC8HNmeM1wdt7f2BpFpJT0sad4LLhuLAkRZe/nAH108fQzymsMtxzrkekatO6WeBSjM7j9RW+5MnsrCk+ZJqJNU0NjbmqKTuvfThDg43J7lhunfbOOeiK5ug3wKMyxivCNpamdkuMzsSjD4KfCbbZYPlHzGzKjOrGjWq904PvKR2K2Wn9eezlcN7bZ3OOdfbsgn65cBkSRMl9QNuARZnziApc5P4JmB1MPwCcI2kUkmlwDVBW+j2HW7mlTWNXD99LDHvtnHORVi3R92YWYuku0gFdBx43MxWSbofqDGzxcCfSboJaAGagDuCZZskfY/UhwXA/WbW1AN/xwn7f6u3c7Ql6UfbOOciT2YWdg1tVFVVWU1NTY+v52tPLqdu615++40v+ha9c67Pk7TCzKo6mlaQvxD69FAzv/7Iu22cc4WhIIN+Wd12mhPG7PNPD7sU55zrcQUZ9NW1W6koHcj5FUPDLsU553pcwQX9noNH+e3HO7nhvLFI3m3jnIu+ggv6F1ZtoyVpzJ7u3TbOucJQcEFfXdvAhBElnFt+WtilOOdcryiooN+1/wivr9vFbO+2cc4VkIIK+udXbSORNG7wbhvnXAEpqKBfUtvAGaMGcfbYIWGX4pxzvaZggr5x3xHeXL+L2dO928Y5V1gKJuif/6CBpOE/knLOFZyCCfpnaxuYPHowZ5V5t41zrrAURNBv33uY5Z80Mfs835p3zhWeggj6pSsbMIMb/JTEzrkCVBBBv6S2galjhjBp9OCwS3HOuV4X+aDfuucQNRt3c6PvhHXOFajIB/3SlQ0AfgFw51zByiroJc2StEbSWkn3djD9v0uqk1Qr6SVJEzKmJSS9F9wWt1+2p1XXNnBu+WlUjhzU26t2zrm80G3QS4oDDwHXAdOAWyVNazfbu0CVmZ0HPA38IGPaITO7ILjdlKO6s7K56SDvbd7jpzxwzhW0bLboLwLWmtl6MzsKLALmZM5gZq+Y2cFg9E2gIrdlnpx0t41fANw5V8iyCfpyYHPGeH3Q1pk7gecyxgdIqpH0pqSbO1pA0vxgnprGxsYsSspOdW0D51cMZdzwkpw9pnPO9TU53Rkr6TagCvhhRvOE4MrkXwJ+LOnM9suZ2SNmVmVmVaNGjcpJLRt3HWDllk/9R1LOuYKXTdBvAcZljFcEbW1Iugr4FnCTmR1Jt5vZluB+PfAqcOEp1Ju16tpUt8313m3jnCtw2QT9cmCypImS+gG3AG2OnpF0IfAwqZDfkdFeKql/MDwSuBSoy1XxXVlS28CM8cMoHzawN1bnnHN5q9ugN7MW4C7gBWA18HMzWyXpfknpo2h+CAwG/r3dYZRnAzWS3gdeAR40sx4P+vWN+6lr2OvdNs45BxRlM5OZLQWWtmu7L2P4qk6Wex2YfioFnowltQ1IcL3/SMo556L5y9jq2gY+O2E4Y4YOCLsU55wLXeSC/uPt+1izfZ+fqdI55wKRC/rqoNvmuuljwi7FOefyQqSC3syort3KxROHM3qId9s45xxELOjXbN/HusYDfrSNc85liFTQV7/fQExw3bnebeOcc2mRCXozY8nKBj535khGDO4fdjnOOZc3IhP0m5sOUb/7oJ+p0jnn2snqB1N9wfgRJdR862qK4gq7FOecyyuRCXqAoSXFYZfgnHN5JzJdN8455zrmQe+ccxEnMwu7hjYkNQIbT+EhRgI7c1ROT+tLtULfqrcv1Qp9q96+VCv0rXpPpdYJZtbhlZvyLuhPlaSa4IpWea8v1Qp9q96+VCv0rXr7Uq3Qt+rtqVq968Y55yLOg9455yIuikH/SNgFnIC+VCv0rXr7Uq3Qt+rtS7VC36q3R2qNXB+9c865tqK4Re+ccy5DZIJe0ixJayStlXRv2PV0RdI4Sa9IqpO0StLdYdfUHUlxSe9Kqg67lu5IGibpaUkfSlot6ZKwa+qMpL8IXgMfSFooKa8upCDpcUk7JH2Q0TZc0jJJHwf3pWHWmNZJrT8MXge1kv5D0rAQS2yjo3ozpt0jySSNzMW6IhH0kuLAQ8B1wDTgVknTwq2qSy3APWY2DZgJfD3P6wW4G1gddhFZ+gnwvJlNBc4nT+uWVA78GVBlZucCceCWcKs6zhPArHZt9wIvmdlk4KVgPB88wfG1LgPONbPzgI+Ab/Z2UV14guPrRdI44BpgU65WFImgBy4C1prZejM7CiwC5oRcU6fMrMHM3gmG95EKovJwq+qcpArgBuDRsGvpjqShwOXAYwBmdtTM9oRaVNeKgIGSioASYGvI9bRhZq8BTe2a5wBPBsNPAjf3Zk2d6ahWM3vRzFqC0TeBil4vrBOdPLcAPwL+CsjZDtSoBH05sDljvJ48Ds5MkiqBC4G3Qi6lKz8m9cJLhlxHNiYCjcCCoKvpUUmDwi6qI2a2Bfg7UltuDcCnZvZiuFVlpczMGoLhbUBZmMWcgK8Cz4VdRFckzQG2mNn7uXzcqAR9nyRpMPAL4M/NbG/Y9XRE0mxgh5mtCLuWLBUBM4B/NLMLgQPkT9dCG0Hf9hxSH06nA4Mk3RZuVSfGUoft5f2he5K+RarL9Kmwa+mMpBLgfwL35fqxoxL0W4BxGeMVQVveklRMKuSfMrNnwq6nC5cCN0n6hFSX2Bcl/Wu4JXWpHqg3s/Q3pKdJBX8+ugrYYGaNZtYMPAN8LuSasrFd0liA4H5HyPV0SdIdwGzgjy2/jyc/k9SH/vvB+60CeEfSKV8bNSpBvxyYLGmipH6kdmgtDrmmTkkSqT7k1Wb2v8Oupytm9k0zqzCzSlLP68tmlrdbnWa2DdgsaUrQdCVQF2JJXdkEzJRUErwmriRPdxy3sxiYGwzPBX4VYi1dkjSLVLfjTWZ2MOx6umJmK81stJlVBu+3emBG8Jo+JZEI+mBny13AC6TeKD83s1XhVtWlS4Evk9o6fi+4XR92URHyp8BTkmqBC4Dvh1tOx4JvHU8D7wArSb0f8+pXnJIWAm8AUyTVS7oTeBC4WtLHpL6VPBhmjWmd1PpTYAiwLHif/SzUIjN0Um/PrCu/v8k455w7VZHYonfOOdc5D3rnnIs4D3rnnIs4D3rnnIs4D3rnnIs4D3rnnIs4D3rnnIs4D3rnnIu4/w9BQkjqCmyC5AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#调用\n",
    "epoch = 5\n",
    "Loss = []\n",
    "Accuracy = []\n",
    "for epoch in range(1, epoch+1):\n",
    "    print(\"start_time\",time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())))\n",
    "    loss, acc = train_runner(model, device, trainloader, optimizer, epoch)\n",
    "    Loss.append(loss)\n",
    "    Accuracy.append(acc)\n",
    "    test_runner(model, device, testloader)\n",
    "    print(\"end_time: \",time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(time.time())),'\\n')\n",
    "\n",
    "print('Finished Training')\n",
    "plt.subplot(2,1,1)\n",
    "plt.plot(Loss)\n",
    "plt.title('Loss')\n",
    "plt.show()\n",
    "plt.subplot(2,1,2)\n",
    "plt.plot(Accuracy)\n",
    "plt.title('Accuracy')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb77c96d-0adc-497a-ad21-31be80893cff",
   "metadata": {},
   "source": [
    "## 3.7 模型保存"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "aaab3d57-6ef0-443e-84ae-95db1f3b74f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "model\n",
    "torch.save(model, './models/model-mine.pth') #保存模型"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93fdf53f-c507-4d12-9f0d-8a1bfc25b388",
   "metadata": {},
   "source": [
    "## 3.8 模型测试"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f07df9f-0265-43f2-b4c1-dce2ffd484d4",
   "metadata": {},
   "source": [
    "下面使用上面训练的模型对一张LED图片进行测试。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "be37046c-5290-4013-92f9-2ba9e75be7ae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAIaElEQVR4nO3dsW4dVRcF4LEdkwpi6vS8QBRKCIrgDdwgIV6BHkWWnyJtFEFA4glSpKBJGhp6REkFMlDGdkxF8Uu5e0feGd11+b+vPTpzZ+710kheOufsXV1dLUCe/W3fAPB6wgmhhBNCCSeEEk4IdaMa3N/fL/+V2/2nd39/c/ZfvXpVzp26cWPzo11cXIyuvbe3V44nfy+d6tm65zo4OCjHLy8vr3VPb6L6vZdl/puv6erq6rVfujcnhBJOCCWcEEo4IZRwQijhhFDCCaH2qu5qb2+vLLYmfV/V9XVz32S80t13p/vs7vpVlzm9dqfrUavf5bfffivn3r59+1r39K+qq+zue/JcbzJ/TXpO2DHCCaGEE0IJJ4QSTgglnBBKOCHUqOfsrLmmctKTTncc7OZPOrdu7qRDXZZ+zWWlW485ufayzDvcXaXnhB0jnBBKOCGUcEIo4YRQwgmh6v0EG12dMalLpkt8qn/rd3O3ebhT99ydaZ2x5rXXXOa39lK6bfDmhFDCCaGEE0IJJ4QSTgglnBBKOCHUqkvGyg8eHqM3nT8xvXbVqX3zzTfl3C+//LIcf/LkSTl+fn5ejn/88ccbx7qtL7tj+DqWjP0vb04IJZwQSjghlHBCKOGEUMIJoYQTQq3acx4eHm4c6/q2NY9s69Ylvnjxohy/c+dOOd4dlffo0aONYycnJ+Xc6brDST88PZZx0g/fvHmzHJ+u0d3mGl49J+wY4YRQwgmhhBNCCSeEEk4IJZwQquw59/f3y/Jnm91Qp+oy33333XLu2dlZOd51alW/uyyzLnHa/072f127S6w++7+81lPPCTtGOCGUcEIo4YRQwgmhhBNClXsZrvlv/e5f4921Ly8vrz3+66+/lnOnz/3gwYNy/PT0dONYt5yte+5p1dIsIRxde3q84cQ2t1K9Lm9OCCWcEEo4IZRwQijhhFDCCaGEE0KNzmybbNPY9Updnzfprd5///1y7tRkWVb33J01+7rJcy3LrAdds79N5c0JoYQTQgknhBJOCCWcEEo4IZRwQqiy55xuR1itTby4uBhduzPZ4rHr1LrOrFuTuc0ucrI95Zrbbnamv1n32dN+eQ3enBBKOCGUcEIo4YRQwgmhhBNCCSeEGu1b25l0mdOOtbr3v/76q5x7dHRUjnf3dn5+Xo6vabLGdlnq7+3rr78eXXvy2dO1orvImxNCCSeEEk4IJZwQSjghlHBCKOGEUKN9a2/cqKdXPec2z0ucrGl8G6q1h2v3dZP9X+/evVvO7Xrt7u+lWlM5Pduz+80Te1JvTgglnBBKOCGUcEIo4YRQwgmh9pplOqM+Y5uVQVWHPH36tJz76aefluNdZdBtjfnBBx9sHPvll19G155u8fjdd99tHDs+Pi7ndvc2sXa9tU1XV1evfThvTgglnBBKOCGUcEIo4YRQwgmhhBNClWt4JkvClqXuMqfX7lQd62effTa69nQ528nJycaxr776qpz7xx9/jD67U/WJ0x5zcozf2v1uIm9OCCWcEEo4IZRwQijhhFDCCaGEE0LFruecdmqT3qv77K6DnWy92d13d3zhrVu3yvHJ97r2lqLV30R335MtP7fNek7YMcIJoYQTQgknhBJOCCWcEEo4IdSo53znnXfKi798+fJ6d/UWVJ3bdD3m9Di5yXF2065x8uzdtbvn7uZX4913ts3vZUrPCTtGOCGUcEIo4YRQwgmhhBNCCSeEKjeP7bqhrsecrOec9lpNfzu6djf/iy++KMcfP35cjk8+e6q6/vQ3m/Sg2+wht8WbE0IJJ4QSTgglnBBKOCGUcEKoVbfG3Oa/xifLsrq6otu+sptfba05WVa1LOtuX/ns2bNy/P79+9e+9rLU994dGdmxNSbw1ggnhBJOCCWcEEo4IZRwQijhhFCz8qix5rKtzuSYvbVVnV333F3f1x1POHF6elqOf/LJJ+V41z1Plqv9F3lzQijhhFDCCaGEE0IJJ4QSTgglnBBq1Z7z4OBg49jaXWPVF3ZdYNfBTtaKdtefrAVdlnkP+u23324c+/zzz8u5ncla0zW33VyWzK03vTkhlHBCKOGEUMIJoYQTQgknhBJOCLXqvrUTVUe6LLOedNppTbvGo6OjjWN///13Obfr67q+77333ivH//zzz41j3ffWdZHd93Z4eFiOV3axx/yXfWthxwgnhBJOCCWcEEo4IZRwQqjYJWPTY/aqf5131+4qga4q6f5t//vvv28c6+qEaSVwdnZWjk+OCOzu7eeff772tXe5Krkub04IJZwQSjghlHBCKOGEUMIJoYQTQo16zsk2jNOtDjtVx9otR+tM76363roO9qeffirHP/zww2vd09vQdZHn5+erXbuziz2oNyeEEk4IJZwQSjghlHBCKOGEUMIJocqtMQ8ODspyaHLs2uQ4uDeZXzk+Pi7Hv//++3J8egRg1fdNtodclvn3Wv2max59uCx1/9ttq7nLbI0JO0Y4IZRwQijhhFDCCaGEE0IJJ4QqF2RO1y1Ousg119/98MMP5Xh1RN+yLMvDhw/L8W696LQvrEy/t+refvzxx3LuvXv3Rp+9Zpe59vrhNXhzQijhhFDCCaGEE0IJJ4QSTgglnBCqXM+5t7e3e5t9vgXTPVI7Vac2XY/Z7Xv70UcflePPnz8vxye6rrF6tu65Osnne1rPCTtGOCGUcEIo4YRQwgmhhBNCqVLCTJc2TSuDSY20zTqiW6Y3rWLWpEqBHSOcEEo4IZRwQijhhFDCCaGEE0KVW2P+v6qOoluWvjPr+r6qy5xu0TjtOZvee/TZncn30v0mtsYE3hrhhFDCCaGEE0IJJ4QSTgglnBCqXM8JbI83J4QSTgglnBBKOCGUcEIo4YRQ/wDsJiYTfH0wVgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "概率： tensor([[7.2506e-11, 7.0065e-18, 7.1749e-06, 7.4855e-13, 7.3532e-08, 8.5405e-17,\n",
      "         2.5753e-15, 9.7887e-10, 2.7855e-05, 9.9996e-01]],\n",
      "       grad_fn=<SoftmaxBackward>)\n",
      "预测类别： 9\n"
     ]
    }
   ],
   "source": [
    "from PIL import Image\n",
    "import numpy as np\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "    model = torch.load('./models/model-mine.pth') #加载模型\n",
    "    model = model.to(device)\n",
    "    model.eval()    #把模型转为test模式\n",
    "    \n",
    "    #读取要预测的图片\n",
    "    # 读取要预测的图片\n",
    "    img = Image.open(\"./images/test_led.jpg\") # 读取图像\n",
    "    #img.show()\n",
    "    plt.imshow(img,cmap=\"gray\") # 显示图片\n",
    "    plt.axis('off') # 不显示坐标轴\n",
    "    plt.show()\n",
    "    \n",
    "    # 导入图片，图片扩展后为[1，1，32，32]\n",
    "    trans = transforms.Compose(\n",
    "        [\n",
    "            #将图片尺寸resize到32x32\n",
    "            transforms.Resize((32,32)),\n",
    "            transforms.ToTensor(),\n",
    "            transforms.Normalize((0.1307,), (0.3081,))\n",
    "        ])\n",
    "    img = trans(img)\n",
    "    img = img.to(device)\n",
    "    img = img.unsqueeze(0)  #图片扩展多一维,因为输入到保存的模型中是4维的[batch_size,通道,长，宽]，而普通图片只有三维，[通道,长，宽]\n",
    "    \n",
    "    # 预测 \n",
    "    output = model(img)\n",
    "    prob = F.softmax(output,dim=1) #prob是10个分类的概率\n",
    "    print(\"概率：\",prob)\n",
    "    value, predicted = torch.max(output.data, 1)\n",
    "    predict = output.argmax(dim=1)\n",
    "    print(\"预测类别：\",predict.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4496c8b-36d5-418d-9a8e-d5bf7fce4d11",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:.conda-pytorch] *",
   "language": "python",
   "name": "conda-env-.conda-pytorch-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
